UCLA C R FE S S al, NATIONAL CENTER FOR RESEARCH ON EVALUATION, 
STANDARDS, AND STUDENT TESTING 


| 4 
NEWYORK CITY DEPAR 
EDUCATION 


Jia Wang, Joan L. Herman, Scott Epstein, Seth Leon, Deborah La Torre, Sandy 
Chang, Velette Bozeman, Wenya Xie, and Julie Haubner 


Copyright © 2019 The Regents of the University of California. 


The work reported herein was supported by grant number 20145515 from the Literacy Design Collaborative with 
funding to the National Center for Research on Evaluation, Standards, and Student Testing (CRESST). 


The findings and opinions expressed in this report are those of the authors and do not necessarily reflect the 
positions or policies of the Literacy Design Collaborative. 


To cite from this report, please use the following as your APA reference: Wang, J., Herman, J. L., Epstein, S., Leon, 
S., La Torre, D., Chang, S., Bozeman, V., Xie, W., & Haubner, J. (2019). Literacy Design Collaborative 2017-2018 
evaluation report for New York City Department of Education. Los Angeles: University of California, Los Angeles, 
National Center for Research on Evaluation, Standards, and Student Testing (CRESST). 


Table of Contents 


EXECUTIVE SUMIMANY iesced occeds foecdecestisecetentdececvecceevecdvcusssacebdedencctivescedesivccevistecedsvecsdbedeedavtewconteetens Vv 
VG TATHOCUCHIO I xixtases cosacoscsecusoradsyccuaadcsscesuasoasas chee ueepebendencnatenmiaasacatenaaaksadt eeeeeeuadadeeaaatalatinneaees 1 
Ted KOBIC-MOGEL secesvsdictde ei eseciead ee dicess goes ae ea sin dade Ho RE oad aR sieves 2 
D2 EValMatiOn: QUESTIONS :cx.casssucecfacttalitocnceutul verstateudadsd alt exeagetes didacs tan ouat dubs ousesds Stele tetacnateasasecans 4 
2.0 Study Methodology isieiccccesviela i sisia vaussanndasevedtids Sis duties duddaadeendeaicaivodensisiastinadaiasesiaeateaesnend 6 
ZA DARA GAINS HOU CIAS sc scascs cownsines Vi saecetl Mnedawer enue aS as eteatee eek ets Manetia enna eaatiad 6 
DD SAUM PIG esses saceee cg dy. e8 ua co tesael xs stu vad aes Sab co Pes wo Eas Ta eed ed as a 9 
2.3 Survey Recruitment and Administration .............cccceccssssssccecceceececeecescseesessesseaaecaeeeeeeeeeeseees 12 
2.4: Module: Scoring PROCESS x1. 22: scasevedid eds thie tease resade Rohs coeeeda dh Cias Pied dua sstet etacd ete s cede snsee tens 13 
2:5:Analytical:Approaches’.s.c.ssec.s2ssersensth usec cokes esau iis eessaieabede cau oebev ieee eens teioee ee hee 14 
4.0 Analyses of LDC CoreTools Data tas cicicscoscsiseieas sovsetisacvisesaventecveanestiessecbenverseenensesseancsnandsacecs 49 
4.1 CoreTools Activity Participation Rates ..........cccccccccccccescesssessssscaeeaeeeceeeeeeseesssssseessessaasaeeees 49 
4.2 Engagement with Key CoreTools ACctivitieS ...........ceccsssececceessseececeeeeeenaecesesssaeeeeeeseseneeeeees 50 
4.3 CoreTools Engagement as an Implementation Variable ............c.ccccccccccccecesssessssssssensteaeeees 54 
AA‘ SUMMArLy: OF RESUIES Ss. 4 6 fo 2c.cessscacesedeossieeestn sted ldaldvess seoessay saves stone ai dace cdgihegsisesdsbadduasbebonaade denees 55 
5.0 Module Artifact Analysis iccccerss sists ceassacndasevedtieds Sot duce dvddcadeendenicadvedendisiaciavasedssssisuateieenns 56 
5.1 Analysis of Elementary MOdUules ..........ccccccccccccccceccecescsecsessenssaeeaeesececceceeessessceeseessaeeaaeaeeeas 57 
5.2 Analysis of Secondary MOdUleS ..........c:cccccscccssccccecceccecesssessesssssesaeeseeeseeeeeeseesseseseessaeeaaaaeeess 58 
5.3 Exploratory Analysis Of MOUIES ...........c:cccccccccecceceecesceessessenesaecaeeeeeeceeeesesseesceessessssaasaeeeas 60 
Bra OUalitative Res UWS cate socius cocdeTusteteal tetvtt deed alevnalsteNgre adyteetussnlanedcavasueadsheaaertnduta evades aacdu sigs 61 
5S -SUMMAry Of RESUIES 2:22.06 ssicsetesnepsaiiek esate baneeesehssessaaauesbevescaelnsbanteieed couseab ous bbaseseatuneniceeoens 63 
6.0 Fidelity of Implementation Analysis ...............sscssccccssssssscecccesssssccececcccessesceccceesssssesceeaneesees 64 
6.1 School- and Program-Level Fidelity of Implementation SCOFeS.............cccceecsscsscsececeeeeeeeeees 64 
6.2 Key Component 1: Common Planning Time for LDC Professional Learning Community 
With Synchronous Coach SUPPOFrt ...........ccccessesssccececceceeeescesessseesseeseaaeeaeeeceeceeeseeesessseesenseaasaeees 66 
6.3 Key Component 2: Asynchronous Support from LDC Coaches. ............ccceseessscsscaeeeeeeeeeeeeees 69 
6.4 Key Component 3: Teacher Implementation Activities ............ccccccccesssessessensecaeeeeeeeeeeeeees 71 
6.5 Key Component 4: Leadership Support at Different Levels................ccccessessscscaeeeeeeeeeeeeees 72 
6:6:SUMIMALYyOLRESUITSS. iiseccssescesebteehias ithe cpslis ons iaaiiess Wik abr eebeetewesalis asibecsdaueeeeea ieee ahaeeeeas 74 
7.0 Student: Outcome Analysis csiisiccs aden cskeds toi vod cal aad coawedusevedvvtatinsteedeaveeveatuasids cavedutssensewdonedenss 76 
7.1 LDC Sample and the Matching Process .0...........cccccccesceessessessensecaeeeececeeeesessesssesseesssesaasaeeees 76 
7.2 Descriptive Results on the Matched Analytic Samples..........c.ccccccccccccccccesseescesssesssstssteeeees 79 
7.3 Outcome Analysis Results: Cohort 1 Middle School Sample ..............ccccceseeessesseessessaaeeeeees 82 
7.4 Outcome Analysis Results: Cohort 2 Elementary Sample ..........cccccccccccceeecessessesseessaaeeeeees 84 
7.4 Outcome Analysis Results: Cohort 2 Middle School Sample ..............ccccceseeessessseseessaaeaeeees 86 
7.5 Summary and Interpretation Of RESUItS ..........ccececcesssessesseecaeececeeceeesessesssesssessessaaeaeees 87 
8.0 SUM Mary OF FIN GIABS :gsscsccccchs cas tadcc cont cecranceope tens uieencearasinacecoues eataniacaeosminasatiaaneeeretantawooeett 90 
8.1 Program Characteristics and Implementation. ..............cccceesssssscsccececceceeeeeeescesssessnsssaaseeeees 90 
8.2 Contextual Factors and IMpleMentation............cccceccesssessessessecaeecececceeesessessseeseessnssaasaeeees 91 
$23: ProgralnirlMPats sx aeccaveecas ceed ons Hgea lect dateden tw esth site Madetiaaa ede laaeed adda sderdelaadvl ans eaeda Hesaaln iets 91 
RETEREICES a5 sestensevccuitect cst eutus nes soucceuocsondendecaluasseivaupesussdavaene mbar sbanii neaduaansbanmtbscaddacaniusne sinitanens 92 
Appendix A: Teacher Survey and ReSPOnses .........cccccssssssececcccsssssceecccecesssceceeccessssesceceeenssssceses 94 


Appendix B: Teacher Leader Survey and R@SpOmses...........ccccccsssssscececcccsssssceccccessssscecceeeessees 111 


Appendix C: Administrator Survey and ReSpOnses. ..........ssccccccsssssseececcccssssececcccesssssceceeeeesssees 125 
Appendix D: LDC Module Rating Dimensions ................sssscccccssssssseececccessssececcccecssescececeeeessees 134 
Appendix E: Principal Interviews............ssccccccsssssssscecccssssesceccccccssscsecescccssssecescccecsseseeeceeeeeesees 140 
WECM OC ss stir crise Scheme onsseas cada biaertneliecsnddt a teastanereanele vlc atucseid ua anthweddulhacGdaues anbcrennaeeuteenideebiacehs 140 
PRESUMES coc acavotienl Guaatcantbaceutaicn cca waboncts caveat nei tuah eugene tadams accra ve bcautnetstaradat etal avast aetng acne, dake Seecuds 143 
Appendix F: Module Artifact Ratings .............ssssssssccscsssssssccccccessssceececccessssecescceacssescesceeaessees 149 
Generalizability Study..........cccccccssssessssscaecceeeeceeeeeecessesssseseseesaeseeseceeeeecesessesseesseenseaeeaeeees 149 
Descriptive ReSults s:..icsrsssvesvinsgcec cls cecesees seeds des sida ssaveckeain cate logy Goes beds sie elann ee teaceeta eevee 152 
Appendix G: Outcome Analysis Methodology ............:sssssscccccssssssceececcccsssssceccccesssssceceeencessees 166 
Analysis Model Specification ..............ccsccssssssssssnsencccceceeceeeeeesessenseasensnaasseeeecsecesceseesseessassssseanes 166 
Student/Teacher Course Exposure WeightiNg ............::ccccsscccessscceessseeceesseeceseeeeessaeeceesseecensess 168 
Caletilation Of BPSCt Si Ze sis cccithdnsnnclhee ats bactecus boas tiie shar wis ulna theupohelueetaidat Ue iewbeesatelathunadstmnobienss 170 


Literacy Design Collaborative 2017-2018 
Evaluation Report for New York City Department 
of Education 


Jia Wang, Joan L. Herman, Scott Epstein, Seth Leon, Deborah La Torre, Sandy Chang, Velette 
Bozeman, Wenya Xie, & Julie Haubner 


CRESST/University of California, Los Angeles 


Executive Summary 


The Literacy Design Collaborative (LDC) was created to support teachers in implementing 
Common Core State Standards (CCSS) and embedding literacy skill development throughout 
content area curriculum. The LDC Investing in Innovation (i3) project focuses on developing 
teacher competencies through job-embedded professional development and the use of 
professional learning communities (PLCs). Teachers work collaboratively with coaches to 
further develop their expertise and design standards-driven, literacy-rich writing assignments 
within their existing curricula across all content areas. 


Engaged in the evaluation of LDC tools since June 2011, UCLA’s National Center for 
Research on Evaluation, Standards, and Student Testing (CRESST) is the independent evaluator 
for LDC’s current federally funded i3 validation grant. CRESST’s evaluation is using multiple data 
sources and a quasi-experimental design to examine LDC implementation and impact in two 
cohorts of schools in the New York City Department of Education (NYCDOE). Study schools 
serve largely Hispanic populations, with a high proportion of students qualifying for free and 
reduced lunch and many English language learners. 


This annual report reflects the second year of implementation in 18 Cohort 1 schools, 
which began implementation in the 2016-2017, and the first year of implementation for 35 
Cohort 2 schools, which commenced at the beginning of 2017-2018. The program experienced 
a high level of attrition among Cohort 1 participants and less than a fifth of the total 2017-2018 
participants are teachers and administrators who participated in LDC in both 2016-2017 and 
2017-2018. With so many new participants and program effects on student learning expected 
only after two years of implementation, this annual report should be considered an interim look 
at LDC progress. It is noteworthy as well that about 40% of participants were at elementary 
schools as that development represents an expansion of LDC’s original focus on middle schools 
and teachers. 


The CRESST evaluation addresses research questions in three major areas: 


e Program Characteristics and Implementation 
e Contextual Factors and Implementation 
e Program Impacts 


The following findings draw on multiple methods. These include surveys of teachers, 
teacher leaders, and administrators; the analysis of LDC modules, which are a central 
manifestation of LDC practice; participant interactions with LDC CoreTools, the electronic 
platform through which teachers access LDC professional development resources (online 
courses, existing LDC modules, module templates, and support for module development); LDC 
administrative records; and administrative data on students and teachers including class 
rosters, student demographics, and student performance on state standards-based 
assessments. We begin with overall findings and then move to summarize perspectives on key 
LDC components, intermediate effects on teachers’ instructional strategies and practice, and 
effects on student outcomes. Detailed evidence with regards to key LDC activities, supports, 
and pedagogical impacts help to explain these positive findings and offer implications for 
further strengthening LDC. 


Overall Findings 


Findings from participant surveys reveal positive results for the LDC intervention, but 
thus far those positive findings on implementation have not been associated with statistically 
significant impacts on student test scores: 


e Teacher, teacher leader, and administrator participants reported positive attitudes 
toward LDC. All measures of satisfaction or improvement were rated positively by 
more than half of the respondents. About two thirds of the teachers expressed 
interest in learning more about how to lead LDC implementation at their schools, and 
about 85% of both teacher leaders and administrators anticipated that their teachers 
would continue with LDC the following year. 


e Participants across all groups perceived a positive impact on student outcomes. 
Approximately three quarters of teachers agreed that LDC helped improve students’ 
reading skills, literacy performance, writing, and content knowledge. Administrator 
respondents were even more likely to report positive impacts on students, with about 
90% reporting impact across different areas of learning. 


e Analysis of student outcomes did not find the impacts of LDC on student learning to 
be significant with p-value at 0.05. The coefficient for our dosage dependent model 
measuring the impact of Cohort 2 middle school teachers was positive and close to, 
but not statistically significant. Coefficients for the other two analyses (of Cohort 1 
returning middle school teachers and Cohort 2 elementary teachers) were very close 
to zero and not significant. 
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Professional Learning Community (PLC) and Teacher Collaboration 


Nearly all LDC teachers participated in LDC-oriented PLCs and 85% reported that their 
PLCs met at least every other week (the frequency expected by the LDC program). For 
teachers and teacher leaders, the most cited barrier to meeting every week was that 
PLC members had other priorities that competed with their participation. 


The frequency with which PLCs met varied across schools and teachers’ individual 
attendance rates also varied greatly. Just under half of schools met attendance goals, 
with many schools experienced challenges related to protecting common planning 
time and ensuring that teacher participants regularly attended PLC meetings. 


Both teacher surveys and PLC reflection forms indicated that PLC meetings typically 
lasted 45 minutes to an hour or an hour or more, generally meeting the LDC standard. 


Teachers valued the collaborative nature of LDC and its PLCs. A large majority of 
teachers credited LDC with making them more likely to collaborate with other 
teachers, not only within their grade levels and content areas but outside of them as 
well. 


About three quarters of teachers reported that they had sufficient time to meet 
during PLC sessions, and over 70% felt that their administrators protected allocated 
resources to ensure that they could meet. However, the most frequently requested 
modification for future years was for more dedicated time during and outside of PLCs 
to plan modules, implement, and receive feedback about implementation. 


LDC Training and Support 


Teachers were nearly uniform in their positive attitudes about the value of their PLC 
participation. They found the PLCs a safe space for sharing instructional plans, 
problem solving, and learning to develop modules. 


Teacher leaders were almost universally reported to be highly approachable, 
supportive, knowledgeable, and helpful, which suggests that programmatic changes 
made for 2017-2018 were successful. (During the 2017-2018 school year, local PLC 
leadership was assigned to the teacher leaders rather than the project liaisons, the 
teacher leaders were provided with a stipend, and principals and assistant principals 
were no longer permitted to play the role.) 


Teacher leaders reported high satisfaction regarding the support they received from 
coaches, professional development offerings, and how the teacher leader role allowed 
them to be instructional leaders in their schools. 


Overall, LDC coaches received positive feedback on the survey, with 99% of teachers 
and 100% of teacher leaders reporting that their coaches gave them appropriate and 
timely feedback and support. Data, however, suggest room for improvement when it 
comes to frequency and usefulness of coach feedback. Analysis of CoreTools analytic 
data revealed that about a quarter of modules did not receive the expected level of 
feedback via comments from coaches, while the peer review functionality was used 
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very infrequently. About one fifth of all teacher survey respondents reported either 
not using written feedback or not finding it useful. 


Most teachers rated CoreTools positively, demonstrating the success of changes LDC 
made prior to the 2017-2018 school year including further refinement of the content, 
sequencing, delivery of CoreTools’ instructional content, and streamlining of 
participants’ learning process. A majority of teachers found the online course 
materials clear, relevant, and useful and, contrary to last year, open-ended responses 
did not reference instructional content as a problem. Almost half of teachers, 
however, rated the ease of use of the online course material as either poor or fair, and 
a similar proportion did not find the videos useful. In addition, analysis of CoreTools 
data revealed that teachers were exposed to a small proportion of the online course 
content during their PLC time, which suggests that there wasn’t sufficient PLC time to 
cover the content LDC intended participants to engage with. 


While 82% of teachers agreed that their school had adequate technology to support 
LDC implementation, many teachers noted in their open-ended responses that there 
were glitches in conducting Zoom meetings. Teachers also suggested the value of 
more in-person time with coaches. 


LDC Implementation 


Teachers reported adapting two existing LDC modules and creating one LDC module 
from scratch on average, in addition to selecting and using mini-tasks outside of the 
modules; the average survey respondent therefore met LDC program expectations 
with regard to creation of modules. According to the survey data, the average teacher 
reported implementing more than two modules in their classrooms. Analysis of 
CoreTools data, however, show that less than 20% of teachers uploaded student work 
for two or more modules (an important proxy for classroom implementation). 


Analysis of program data suggests that while nearly all participants were engaging 
with the module-building platform, the level of engagement did vary greatly across 
individuals and across subgroups (role, cohort, school level, content area) as 
evidenced by the large variation in the number of views, edits, and comments. 
Analysis of module components found that two thirds of teachers failed to engage at a 
basic level by editing the teaching task, while a third of teachers met the fidelity 
threshold by editing multiple key portions of modules. 


The majority of teachers (75% to 92%) reported success in nine key areas of LDC 
module development. Teachers were most confident in selecting focus standards, 
creating the writing assignment, identifying skills needed in the module, and making 
writing assignments relevant and engaging. The module analysis, however, suggests 
that the materials adapted and created by PLC members varied in levels of completion 
and quality. and 


With regards to their classroom implementation of LDC modules, the majority of 
teachers reported success with all six key areas queried (81% to 88%). Teachers were 
most confident with engaging students in complex text, locating evidence of standards 


viii 


in final student work, and engaging students in understanding the assignment and 
rubric. Despite teachers’ reported confidence, if we use uploaded student work as an 
indicator, it appears that a majority of modules might not have been implemented in 
the classroom. 


Overall, module quality increased substantially from 2016-2017 to 2017-2018, 
particularly among elementary level modules. Looking just at 2017-2018 modules, 
elementary modules were on average higher quality than secondary modules. 


An exploratory look at module quality over time for teachers remaining in the 
program suggests some improvement from the prior year. 


Leadership Support 


Almost all teachers and teacher leaders reported that their administrators encouraged 
LDC participation at the school. The majority of teachers and teacher leaders agreed 
with administrators that they allocated resources to ensure that LDC teachers could 
participate in meetings. Administrators generally voiced strong support for LDC, but 
there were differences with how active teachers and teacher leaders viewed that 
support. LDC may want to understand the differences in these perceptions better. 


There was great variation across schools in the extent to which administrators and 
teacher leaders took advantage of in-person professional development opportunities, 
and also the frequency of teacher leader/coach planning calls. Most schools did not 
meet fidelity thresholds on these indicators. 


Compared to uniformly positive ratings by school administrators on most items, their 
ratings of district support were relatively low. This too signals an area where LDC may 
need to strengthen its efforts. 


Impact on Teacher Practice 


The majority of teachers reported improving their practice in seven LDC-related skills 
(73% to 85%). Teachers felt they learned most in selecting focus standards, creating 
standards-driven writing assignments, and identifying skills students need in writing 
assignments (skills concentrated at the beginning of the LDC learning cycle). 


Three quarters or more of teachers agreed that participating in LDC raised their 
expectations for students’ writing, helped them incorporate writing assignments into 
their existing curriculum, and made them more likely to collaborate with other 
teachers on designing instruction. 


Subgroup Effects 


CoreTools analyses showed that teachers in Cohort 1 schools, both returning and new, 
engaged with the online platform at higher levels than teachers in Cohort 2 schools. Among the 
rated elementary modules, modules by Cohort 1 teachers scored higher on four out of the six 
dimensions. Among rated secondary modules, modules by Cohort 2 teachers had the highest 
scores. Survey results across multiple dimensions revealed on average more positive attitudes 
by Cohort 1 returning teachers than Cohort 1 new teachers and Cohort 2 teachers. Cohort 1 


returning teachers found asynchronous support more helpful, reported producing greater 
numbers of modules, were more confident in their ability to design high quality LDC instruction, 
reported greater support from administrators, felt more empowered regarding their role in LDC 
implementation and expansion, and were more likely to report impact on both their own 
practice and student learning. Given that they were in their second year of implementation, 
many of these differences are not surprising and indeed welcome. Results for new participants 
in Cohort 1 schools highlight some challenges. These teachers were less likely to find their PLC 
to be effective and to perceive their administrator to be supportive of LDC. 


Final Conclusions 


LDC’s second year of implementation shows the results of its continuous improvement 
efforts and a positive trajectory toward program success. Changes made based on last year’s 
evaluation results—such as the further streamlining and refining of online courses and the 
changing of the school liaison role to that of a teacher leader—have paid off in greater 
participant satisfaction and greater fidelity to the program’s logic model. 


Evidence suggest that second year implementation was largely faithful to the structure of 
LDC’s implementation model, although in many cases individual teachers and schools were not 
meeting implementation goals. Most PLCs met at least every other week as they were expected 
to do, but a sizable minority struggled to protect common planning time and ensure high 
attendance rates. Participants engaged with CoreTools resources, but to varying degrees. 
Coaches provided feedback through multiple mechanisms, and while PLC members reported 
appreciating this feedback, feedback was not uniform and not all teachers found it useful. 
Teachers also reported meeting LDC expectations for module development and 
implementation, although data from the CoreTools analyses suggest that, for many teachers, 
module development and implementation may not have been fully complete. 


The data from all respondents’ surveys, coupled with the analysis of module quality, 
suggest that LDC is benefiting most teachers’ pedagogical skills and particularly their ability to 
integrate standards-based writing assignments and assessment into their curriculum. 
Respondents also perceive LDC to have positive results on student learning—particularly with 
regard to literacy skills and writing, although thus far exploratory quasi-experimental analyses 
have not revealed a statistically significant impact of LDC on student scores on state ELA 
assessments. 


This year’s results revealed some areas of challenge. While the majority of participants 
were Satisfied and positive about the benefits of LDC, sizeable proportions were not ina 
number of areas. For example, while the great majority of teachers were confident and noted 
improvement in areas related to creating standards-based teaching and writing tasks, and 
evaluating them, sizable minorities were less confident about the specifics of planning lessons 
and practicing formative assessment. LDC should consider what can be done do to strengthen 


their efforts in these areas and/or target further follow-up support for teachers during the 
coming year. 


Adequate time was a frequently cited concern. Respondents voiced that they hoped for 
more collaborative time, more prep time, time before the start of the year, and/or 
compensated time. Technology also remained a problem for some participants. While some 
respondents were frustrated with technical problems that marred the experience of Zoom 
meetings, others would have simply preferred in-person rather than digital meetings with 
coaches. Participants also seem to have made relatively limited use of the CoreTools 
interactivity, for example with regard to commenting and revision. Some participants also 
reported frustration with CoreTools usability and the utility of its video resources. 


Survey responses also suggest that leadership support needs continuing attention. 
Although teachers and teacher leaders were generally positive about their school leaders’ 
knowledge of and support for LDC, these leaders did not seem to meet LDC’s expectations for 
being involved in program implementation by observing and providing feedback to participants. 
Similarly, administrator responses suggested that district support was a relative weakness. 


As we noted, this report presents interim findings, before most participants have had the 
two years of experience with the program that LDC believes they need to meet program goals. 
The findings at this point are definitely positive. The 2018-2019 school year will provide that 
crucial second year of experience and, barring unexpected attrition, strong data for fully 
examining the implementation and effects of LDC’s i3 implementation, and the factors that 
contribute to LDC success. 
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1.0 Introduction 


The Literacy Design Collaborative (LDC) was created to support teachers in implementing 
Common Core State Standards (CCSS) by integrating literacy skill development throughout the 
content areas. The LDC Investing in Innovation (i3) project focuses on developing teacher 
competencies through job-embedded professional development and the use of professional 
learning communities (PLCs). Teachers work collaboratively with coaches to further develop 
their expertise and design standards-driven, literacy-rich writing assignments within their 
existing curriculum across all content areas. LDC is a national community of educators providing 
a teacher-designed and research-based framework, online tools, and resources for creating 
both literacy-rich assignments and courses across content areas. Used by individual teachers, 
schools, and districts in 40 states for the past four years, LDC also is a statewide adopted 
strategy for Common Core implementation in Kentucky, Colorado, Louisiana, and Georgia. 


UCLA’s National Center for Research on Evaluation, Standards, and Student Testing 
(CRESST), in collaboration with its partner Research for Action (RFA), engaged in the evaluation 
of LDC implementation and its impact on student learning and teacher effectiveness starting in 
June 2011, via two parallel research studies funded by the Bill and Melinda Gates Foundation. 
Those studies included an examination of LDC implementation in eighth grade social studies 
and science classrooms in Kentucky and Pennsylvania and a districtwide implementation in 
sixth grade advanced reading classes in a large district in Florida. Results for the studies are 
available in two CRESST reports (Herman et al., 2015a; Herman et al., 2015b), as well as a 
journal article published by AERA Open (Herman, Epstein, & Leon, 2016). 


Currently, CRESST serves as the independent evaluator for LDC’s federally funded 
Investing in Innovation (i3) validation grant. The LDC i3 study is examining the implementation 
and impact of LDC in two large urban school districts: New York City Department of Education 
(NYCDOE) and a large school district on the West Coast (the former of which this report is 
focused on). The evaluation study is a comprehensive mixed-methods evaluation to understand 
the impact of LDC on student learning using a quasi-experimental design, as well as to 


document LDC’s impact on teacher skills and practices. Specifically, the evaluation study 
addresses a rich range of questions about program characteristics, conditions, and impacts in 
the context of two large urban school districts. The study draws on data from two cohorts of 
schools, with each school housing a PLC of teachers who engage in professional learning about 
LDC and implement LDC mini-tasks and modules in their classrooms. The study measures 
teacher implementation and skill improvement with teacher surveys, analytic data from LDC’s 
online CoreTools module building platform, artifact analysis, and LDC administrative records. 
While we document the core strategies of the LDC model as implemented and provide support 
for LDC improvement, the central focus of our comprehensive mixed-methods evaluation is 
examining the impact of LDC on teacher practices and student learning using a quasi- 
experimental design. 


The first i3 evaluation cohort of schools began implementing LDC during the 2016-2017 
school year, and the second cohort of schools began implementing LDC during the 2017-2018 
school year. This annual progress report examines LDC implementation during the 2017-2018 
school year in NYCDOE. A parallel progress report focuses on implementation in a large district 
on the West Coast (Wang, et al, 2019). The current annual progress report presents results 
from (a) surveys of LDC classroom teachers, teacher leaders, and school administrators; (b) 
analyses describing how LDC participants interacted with the CoreTools module building 
platform; (c) rating by CRESST of instructional modules created by LDC participants; (d) analysis 
of the fidelity of implementation across multiple key components, indicators, and data sources; 
and (e) student outcome analyses using the quasi-experimental design. 


Survey results provide a window into how LDC was implemented in 2017-2018, the 
perceived utility and effectiveness of various program components, and the perceived impact 
of LDC on both teacher and student skills and knowledge. The analysis of CoreTools user data 
and modules created and adapted by teachers provide evidence on the level of engagement 
with the online platform and module design process, and the quality of the products created by 
the teachers and PLCs. Fidelity of implementation analysis, utilizing the fidelity matrix designed 
collaboratively by LDC and CRESST, provides a broad picture of how schools and the program 
performed on key fidelity indicators in 2017-2018. Exploratory analyses of the effectiveness of 
LDC in increasing student learning for Cohort 1 schools with two years of LDC implementation 
and Cohort 2 schools with one year of implementation are also included in the report. The 
confirmatory analysis will be conducted next year by pooling data on teachers from both 
Cohort 1 and 2 schools who implemented LDC in two consecutive years and their students. 


1.1 Logic Model 


The logic model includes four key intervention components that are predicted to be the 
drivers of change in teacher practice and student learning (see Figure 1.1). These are a coach- 
supported Professional Learning Community (PLC) formed to implement the LDC intervention 
at the school site and provide a space for teacher collaboration; asynchronous support from 


coaches primarily in the form of feedback in CoreTools through comments and peer review; 
implementation activities completed by participating teachers including module development 
and classroom implementation; and leadership support at different levels. Note that the model 
also indicates LDC’s expectations for the level of implementation in each area. 


LITERACY DESIGN COLLABORATIVE i3 LOGIC MODEL 


MEDIATORS 


Activities Participation Short term Medium term Long term 
Common Plannin Student Increased Students 
timetor LDC 9 - PLC members participate in at least 60 engagement CCRS skills acquire 
Professional minutes of common planning time during knowledge and 
Learning each consecutive two weeks. a ete sli to st 
Community (PLC) - Digital coach joins PLC every other week. ‘est scores college ani 


Sequencing incorporates blocks from LDC 
online courses. 


i career read 
with synchronous ¥ 


coach support 


Higher rates 
of course Educational 
completion attainment 


Increased teacher 
expertise and skill 


increases 
Asynchronous : - development 
Coach provides feedback on modules in 
support from LDC K Higher 
coaches CoreTools via comments and peer review H 
rubric ratings. a uation 
rates 


Increased labor 
market 


Teacher - Teachers develop at least one module per outcomes 
implementation year in Years 1-2. 
activities Teachers teach at least two modules per year 

in Years 1-3 


More effective 
CCRS-aligned 
instruction 
and use of 
formative 
assessment 


District instructional specialists visit schools 
three times per year to sit inon PLC, meet with 
principal and teacher leader, and observe 
instruction. 
Principals participate in LDC by protecting PLC 
time, setting goals for LDC work, facilitating & 
participating in school visits from district 
instructional specialists, and observing LDC 
instruction. 

Teacher leaders facilitate PLC on the ground, 
collaborate with administrator, and 
communicate regularly with coach. 


Leadership 
support at 


different levels 


Contextual Factors: District support, school and student demographics, other policies and initiatives 


Figure 1.1. LDC i3 logic model. 


The logic model predicts that the four key components will lead to increased teacher 
expertise and skill development, and more effective Common Core aligned instruction which 
incorporates ongoing formative assessment. In turn, increased teacher capacity and more 
effective instruction will lead to increased student engagement in the short term; increased 
student skill acquisition, higher test scores, and higher rates of course completion in the 
medium term; and improved college and career readiness, education attainment, graduation 
rates, and labor market outcomes in the long term. 


Note that the logic model was revised based on refinements to the program in response 
to learning from the pilot year (2015-2016) and the first year of Cohort 1’s implementation 
(2016-2017). 


1.2 Evaluation Questions 


Our evaluation questions focus on addressing three main areas: program characteristics 
and implementation, contextual factors and implementation, and program impacts. This 


progress report provides findings on many, but not all, of the evaluation questions. In 


particular, there is limited information available regarding program impacts. This report 
provides an exploratory look at how the refined LDC model is impacting the learning of students 
under Cohort 1 teachers in their second year of implementation and Cohort 2 teachers in their 
first year of implementation. The confirmatory analysis will be conducted after year 2 data are 
available for both cohorts of schools/teachers; that analysis will first be shared in next year’s 
annual report. 


1. Program Characteristics and Implementation 


a. 


Who are the participating teachers and schools? Are they representative of the 
teacher/school populations of the respective district on years of teaching, 
education level, prior student performance, etc.? 


How is the LDC program implemented in each district? What are the core 
components (e.g., training, tools, on-site or other direct support) and who are the 
key participants? In what ways did the LDC implementation align with the 
intended model? 


In what ways do teachers implement the LDC tools in their classrooms? To what 
extent do teacher practices align with intended LDC practices? 


How are teachers utilizing the online LDC system (including online tools, 
exemplars, collaborative work spaces, and technical assistance) in terms of 
frequency and use of key features? Does this vary by teacher characteristics? What 
are teachers’ perceptions of the value and quality of the online LDC system? 


What types of LDC professional development opportunities are offered to and 
utilized by teachers at each school/district? Are teachers and schools satisfied with 
the LDC professional development opportunities they received? 


2. Contextual Factors and Implementation 


a. 


What factors facilitate or hinder successful implementation of the LDC model at 
the teacher, school, and district levels? 


How can implementation of the model be improved at the teacher, school, and 
district levels? 


What other educational reforms are being implemented in the participating 
schools and districts? What are their influences on the LDC adoption in the schools 
and districts? Are schools able to align reform efforts? 


What are the roles of school and district leadership in shaping the LDC 
implementation? 


3. Program Impacts 


a. 


What is the impact of LDC on the academic performance of participating students 
as measured by the state assessments? 


Do the academic impacts vary by student subgroup including prior achievement, 
race, ethnicity, socioeconomic status, gender, language proficiency, and/or 
disability? Does LDC help close the achievement gap between student subgroups? 


Do the academic impacts vary by student grade level or subject? 


What is the impact of LDC on teacher skill improvement and learning as measured 
by CoreTools and by the quality of LDC modules they produce? What is the self- 
reported impact of LDC on teacher learning? 


To what extent do teachers report changes in their practice (e.g., teaching 
strategy, collaboration with others) and changes in their comfort in implementing 
CCSS during and after the LDC intervention? 


What is the relationship between the fidelity of implementation, fidelity of 
intervention, and student learning? What are the conditions and contexts under 
which LDC tool use is most effective? 


To what extent do Cohort 1 participating schools and teachers continue their LDC- 
influenced practices in the 2019-2020 school year after the LDC support ends? 
What contributed to their decision to continue or stop? What factors contributed 
to their levels of continued implementation? How does Cohort 1’s actions align 
with their previously stated intentions for continuation of LDC-influenced practices 
as reported in spring 2017? To what extent do Cohort 2 participating schools and 
teachers plan to continue their LDC-influenced practices after the LDC support 
ends? 


2.0 Study Methodology 


In this chapter, we provide an overview of the methodology we used. We begin by 
describing the various instruments and data sources we used for the study, including surveys of 
classroom teachers and teacher leaders participating in PLCs and administrators overseeing the 
implementation; analytic data from LDC’s CoreTools platform; module artifacts including 
samples of student work; LDC administrative data; and administrative data on students and 
teachers used for outcomes analyses. We then describe the sample of educators and schools 
for each of these data sources. Finally, we discuss the methodological approaches for the 
various analyses we conducted. 


2.1 Data and Instruments 


We next describe each of the data instruments and the elements they contain. Most 
variables are measured at the teacher level, which is the unit at which the LDC intervention is 
being implemented. Administrative data for the analysis of the impact of LDC on student 
learning includes school, teacher, and student level variables. 


Surveys (teachers, teacher leaders, and administrators). In 2017-2018, four different 
surveys were administered to LDC participants playing three different roles: teacher, teacher 
leader, and administrator. Some teacher leaders were classroom teachers who implemented 
LDC with their students, while other teacher leaders were out of classroom faculty (coaches or 
coordinators); these two groups received different versions of the survey tailored to their roles. 
Thus, four versions of the surveys were administered in spring 2018: (1) teacher, (2) teacher 
leader (for coaches and coordinators), (3) teacher leader (for teachers), and (4) administrator. 


In collaboration with LDC, CRESST made minor revisions to the 2016-2017 surveys. These 
refinements involved streamlining the language of certain questions, in particular those 
capturing teacher pedagogical practice and the perceived impacts of LDC on teachers and 
students. In a few cases, questions and items were also added to collect systematic information 
on program conditions and impacts that were observed by LDC and CRESST anecdotally. 


The surveys were designed to capture multiple perspectives on key aspects of LDC’s logic 
model (see Figure 1.1),1 and to provide data to answer the evaluation’s research questions 
presented earlier. Survey questions targeted at the three roles fall under the domains and 
subdomains in Table 2.1. Domains were selected to align with the LDC i3 logic model and with 
the CRESST evaluation questions. Note that most domains cover multiple subdomains, 
constructs, and survey questions. Professional Learning Community/Teacher Collaboration, for 
example, captures the intensity, frequency, and collaborative environment of common planning 
time; LDC Training and Support includes quality of online courses, utility and effectiveness of 


1The survey domains were aligned to this version of the logic model for the pilot year. The logic model has since 
been revised to align with the revised LDC implementation plan. 


coach support, etc.; and LDC Implementation encompasses module creation, classroom 
implementation of modules, and module peer review. 


Table 2.1 


Survey Domains for Three Respondent Groups 


Domain Teacher dearer Administrator 
Leader 

LDC Participation X X X 
Professional Learning Community and Teacher Collaboration X X X 
LDC Training and Support X X X 
LDC Implementation 

Module Creation X X 

Classroom Implementation X 

Module Peer Review X 

Alignment X X 
Leadership Support 

Teacher Leader Support X 

School Administrator Support / Classroom Observation X X X 

Teacher Leader Leadership Role X X X 

District Support X X 
Impact 

Impact on Teacher Practice and Learning X X 

Impact on Student Learning X X 
Scale-Up and Sustainability X X 
Facilitators and Barriers X 
Areas of Improvement X X X 


Teachers and administrators were asked to reflect on both LDC’s Impact on Teacher 
Practice and Learning and Impact on Student Learning. Questions within a number of domains 
further asked respondents to reflect on conditions and supports that may potentially impact 
LDC’s implementation. These domains included teachers’ perceptions of Facilitators and 
Barriers to implementation and perceptions regarding leadership roles and support for LDC at 
different levels. Teacher leaders and administrators were also asked for their perceptions 
regarding if and how LDC would be sustained and expanded within the school. Finally, all 
respondents were asked open-ended questions regarding Areas of Improvement for LDC 


implementation. Teacher, teacher leader, and administrator surveys can be found in 
Appendices A, B, and C. 


LDC CoreTools. The CRESST team received the LDC program data on i3 participants’ 
interactions with the CoreTools module-building platform. The data files captured a number of 
behaviors including document page viewing, document editing, document commenting, coach 
use of peer review, exposure to instructional content found in the “Learn” tab in CoreTools, and 
uploading of student work. The data contained date- and time-stamped records of participants’ 
activities in all of these areas. 


Using these data files, we were able to analyze variation in the number of times 
participants viewed, edited, and commented on documents across the school year. We 
generated descriptive statistics (minimum, maximum, mean, and standard deviation) for the 
number of times participants viewed a document page, edited a module document, and 
commented on a module document. We then produced descriptive statistics on these 
behaviors for each role group (teacher, teacher leader, and administrator), school level 
(elementary, K-8, middle, 6-12, and high), cohort, and content area subgroups. We also 
compared the average engagement in these key activities across two groups: teachers who 
completed and taught LDC modules (as indicated by the upload of student work samples) and 
those teachers for whom there wasn’t evidence of module implementation. The results on 
these metrics of engagement are reported in Chapter 4. 


CoreTools data were also used to analyze to what extent: teachers were exposed to 
instructional cycles of LEARN content; coaches provided feedback via commenting and peer 
review functions; teachers edited key portions of modules; and teachers uploaded student 
work (a proxy for classroom implementation). Results for these indicators are reported in 
Chapter 6. 


Modules. Our existing module rating rubrics (Reisman, Herman, Luskin, & Epstein, 2013) 
were designed to examine the instructional quality and coherence of the LDC modules, and to 
address the rigor in both content and literacy development materials (i.e., template task, 
student work samples, and descriptions of the pacing and goals of the modules). The first two 
dimensions examined the quality of the teaching task, while the remaining four dimensions 
focused more holistically on module quality: 1) effective writing task; 2) alignment to the CCSS 
and local and state literacy and content standards; 3) fidelity to LDC module instruction; 4) 
quality instructional strategies; 5) coherence and clarity of module; and, 6) overall impression. 
Each of the dimensions was rated using a five-point scale with anchor points on the first five 
dimensions ranging from not present or realized to fully present or realized and the final 
dimension ranging from inadequate to advanced LDC module implementation. For each 
dimension, a one represented the lowest possible level of quality, while a five represented the 
highest level of quality (see Appendix D for the scoring rubrics). 


LDC Administrative Records. The fidelity matrix analyses in Chapter 6 utilize 
administrative records collected by LDC as part of their ongoing program management. These 
records include (1) PLC Reflection forms, and (2) attendance records for in-person meetings 
organized for the benefit of school administrators and teacher leaders. The PLC Reflection form 
captures attendance at regular PLC meetings, the amount of time spent on LDC during these 
meetings, whether teacher leaders had a separate planning call with coaches, and the subjects 
that PLCs were working on during sessions. 


Administrative data used in student outcomes analysis. Student-level variables utilized in 
the outcome analysis included race/ethnicity, gender, poverty status, special education status, 
English language proficiency, grade, and prior achievement in mathematics and ELA and 
outcome year achievement in ELA on state assessments. The analysis also utilized a measure of 
years of teaching experience. We also requested and received roster files that establish a link 
between teachers and students via specific courses. 


2.2 Sample 


Twenty Cohort 1 schools began implementing the LDC program in the 2016-2017 school 
year, with 219 classroom teachers participating and 47 administrators overseeing the work (see 
Table 2.2). The 29 Cohort 1 schools included two elementary schools, four K-8, 17 middle, three 
6-12, and three high schools. The program, however, experienced a high level of attrition from 
2016-2017 to 2017-2018, as summarized in Table 2.2. Over a third of schools dropped out of 
the program after 2016-2017, and there was also substantial attrition within the remaining 
schools. In total over three quarters of the teachers participating in LDC in 2016-2017 were no 
longer participating in 2017-2018. 


Table 2.2 


Attrition of Cohort 1 Teachers, Administrators, and Schools 


Teachers Administrators Schools 
Participated in LDC in 2016-2017 219 47 29 
Dropped out of LDC after 2016-2017 166 25 11 
Remained in LDC in 2017-2018 52 22 18 
Attrition Rate 76% 53% 38% 


To better understand teacher retention rates at LDC schools after the 2016-2017 school 
year, we conducted an interview study of school administrators in both NYCDOE and the West 
Coast district we are also studying. Twenty principals and assistant principals volunteered to be 
interviewed. The principal interviews reveal the diversity of reasons behind schools’ varying 
success in retaining teachers. The main factor in explaining variation in teacher retention was 
teacher and grade-level team decisions to leave or stay, followed by the principals’ decision to 


switch participation of teachers from year 1 to year 2. Teacher buy-in also seemed to affect 
teachers’ decisions on whether to continue. Appendix E described the results in detail as well as 
the methodology for the study. 


A new cohort of schools started in the 2017-2018 school year. Cohort 2 included 35 
schools new to LDC in 2017-2018, which were comprised of 16 elementary schools, 11 middle 
schools, five K-8 schools, and three 6-12 schools. Because of the high level of attrition in Cohort 
1 and the greater number of recruited schools in Cohort 2, two thirds of the participating 
teachers and administrators in 2017-2018 were from Cohort 2. There was also a group of 68 
new participants who joined existing Cohort 1 PLCs in 2017-2018. Table 2.3 shows the number 
of participants in 2017-2018 by cohort category and school level. 


Table 2.3 
Number of Participants by Cohort and School Level, 2017-2018 


Total 

Elementary K-8 Middle 6-12 High Participants 
Cohort 1 Returning Participant 10 10 40 9 5 74 
Cohort 1 New Participant 10 6 41 9 2 68 
Cohort 2 Participant 147 34 79 23 0 283 
Total Participants 167 50 160 41 7 425 


In addition to the CoreTools analytic files, we also received module artifacts from LDC for 
an analysis of the quality of module design. We restricted our analysis to modules that were 
created during the 2017-2018 school year and included original uploaded student work 
samples, because these samples were considered a proxy for implementation in the classroom 
and were required for scoring modules. That restriction yielded a pool of 307 modules that 
were authored or coauthored by 249 teachers (over half of all participating teachers), across 49 
schools. To align with our research questions and to have analyzed modules be representative 
of the teacher sample, we sampled the last module created by each teacher wherein they 
served as the sole or lead author. Our intent was to represent teachers’ best LDC work, and 
presumably modules created later in the school year would be more skilled than those created 
earlier. The total sample was 141 modules. Table 2.4 provides further detail about the 
distribution of modules. We also conducted an exploratory analysis of the five elementary 
teachers and 14 secondary teachers who created a complete module both this year and during 
the previous school year. 
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Table 2.4 
Distribution of Modules Rated by Grade Band, Cohort, and Subject 


Elementary Secondary Total 
Variables # % # % # % 

Cohort 

Cohort 1 Returning 10 15.38 22 28.95 32 22.70 

Cohort 1 New 2 3.07 21 27.63 22 15.60 

Cohort 2 54 83.07 33 43.42 87 61.70 
Subject 

ELA 34 52.31 44 57.89 78 55.32 

Science 9 13.85 9 11.84 18 12.76 

Social Studies 22 33.85 23 30.26 45 31.91 
Total 65 46.10 76 53.90 141 100.00 


We now turn to Table 2.5, which displays the sample sizes for the different study 
measures and analyses included in this report. The population includes 2017-2018 participants 
in all three cohort categories (Cohort 1 returning participant, Cohort 1 new participant, and 
Cohort 2 participant). As can be seen, data were available for a majority of the 2017-2018 
participants for each of the measures and analyses. Sixty-nine percent of teachers consented to 
participate in the study, with 67% of all teachers completing the survey in spring 2018. The 
consent rate for administrators (79%) was higher than that of teachers while the survey 
response rate for administrators (28%) was much lower than the teacher rate. Nearly all 
teachers (91%) and most administrators (73%) were present in the CoreTools dataset, which 
was delivered to CRESST directly by LDC and did not depend on teachers’ individual study 
consents. Almost three quarters of teachers were associated with the modules we studied in 
our artifact analysis. 
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Table 2.5 


Number of Participating Schools and Teachers in the District and Sample Size for Different Instruments 


Number of Number of schools in 
classroom Number of which teachers and 
teachers administrators administrators practice 
Participated in LDC 345 80 53 
Consented to CRESST study 237 63 49 
Completed survey 231 22 47 
Present in CoreTools dataset 315 58 51 
Author/coauthor of module with student 249 1 49 
work uploaded 
Included in outcomes analysis 111 N/A 34 


The school district required individually signed consent forms before releasing teacher data and 
teacher/student rosters, so for the outcome analysis, we only received data on teachers who consented 
to participate in the study. As described later, there was sufficient data to be able to conduct three 
exploratory quasi-experimental analyses for this report, testing the impact of returning Cohort 1 returning 
middle school teachers, Cohort 2 elementary school teachers, and Cohort 2 middle school teachers. The 
final row in Table 2.5 reflects the number of unique teachers and schools that were represented in these 
three analyses. There was insufficient student sample size to conduct analyses of the impact of Cohort 1 
new participants or Cohort 1 returning elementary school participants, so the teachers in those groups 
were not included in analyses. 


The sample was further restricted by the need for student achievement data for both the outcome 
year (2017-2018) and the baseline year (2015-2016). As a result, participants teaching either in high school 
or lower elementary grades (K-3) could not be included in the student outcome analyses. Middle school 
teachers not teaching a core ELA, social studies/history, or science class also were not included in the 
analyses. Overall, due to all these restrictions, the three outcome analyses are based together on a little 
under a third of the teachers that were participating in LDC in 2017-2018. 


2.3 Survey Recruitment and Administration 


As noted above, one third of the participant sample in 2017-2018 was made up of 
teachers and administrators from Cohort 1 schools; many of the consent forms from those 
participants had successfully been collected by CRESST in 2016-2017.Consent forms were 
collected in person, at PLC meetings via video conference, and over emails. The consent forms 
included language stating that the study was voluntary, all data would be protected, and that 
by signing the form, participants gave their consent to be emailed an electronic survey in spring 
2018 and their permission for CRESST to request district data that link the teacher to students. 
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Directly compensating educators for participation in research is not permitted in New 
York State. We therefore offered instead gift cards of $300 to the school to be used for school 
supplies to directly benefit students. The gift cards were given to the school if the school’s 
teacher leader and at least five teachers participating in the PLC completed the survey. 


Surveys were administered in spring 2018. CRESST coordinated with LDC staff and coaches 
to administer the online surveys during PLC sessions. Multiple email reminders were sent to 
participants who were not present at the common planning session or who otherwise did not 
complete the survey. The teacher survey was closed at the end of the school year in June 2018. 
Administrator surveys were left open until October of the new school year, with LDC staff 
conducting outreach to administrators in an effort to boost response rates. 


2.4 Module Scoring Process 


LDC requirements specified that all teachers implement at least two modules over the 
course of the year, with the first spanning one week based on one text and the second 
spanning across multiple weeks and incorporating multiple texts. Modules could be developed 
as original units of work or could be adapted from existing modules within the LDC CoreTools 
Library. Modules could also be either developed or adapted in solo or collaboratively with other 
teachers within the PLC. 


LDC modules were collected from elementary and secondary teachers who participated in 
LDC during the 2017-2018 school year. As stated previously, to align with our research 
questions and to have the modules be representative of the teacher sample, it was decided to 
rate the last module created by a teacher (as a first author) that included the required 
materials. In total, 141 modules created by NYCDOE teachers were rated with 46 or 32.6% 
being rated by two panelists. Further details about the modules can be found in Appendix F. 


Seven expert raters with experience teaching in the targeted grade spans and content 
areas were recruited from Los Angeles county schools. Four panels were convened with two 
experts rating each of the following sets of modules: elementary ELA and social science, 
elementary and secondary science, secondary ELA, and secondary social science. One of the 
seven expert rater served on both the secondary ELA and the secondary social science panels 
because of her wide range of experience. 


Separate trainings lasting approximately two hours were conducted for each module 
scoring panel. All trainings were conducted by a member of the evaluation team who is an 
expert on the Common Core and the rating of student and teacher artifacts. The training 
included an overview of the LDC goals, template task, the structure of the modules, and the 
CRESST rating dimensions. Once the training was complete, calibration was conducted by 
having teachers individually score and then discuss their ratings for one module in the content 
area on which they would be focusing. Scoring was then conducted on subsequent days with 
each module individually rated. Those modules rated by two expert teachers were then 
discussed with the goal, but not the requirement to reach consensus (see Carlson & McCaslin, 


13 


2003). All discussions were facilitated by the same evaluation team member who conducted 
the initial training. 


2.5 Analytical Approaches 


Both quantitative and qualitative analytic methodologies were applied to the data to 
answer the evaluation questions about how LDC was implemented, conditions affecting 
implementation, and program impacts. The following describes the approaches used to analyze 
each data set. 


Surveys. Survey responses were analyzed using descriptive statistics for multiple-choice 
items and qualitative coding for open-ended responses. As previously noted, surveys were 
administered to teachers, teacher leaders, and administrators. Some teacher leaders were 
classroom teachers while others were coaches or coordinators. Because of this, the samples 
were not mutually exclusive, with teacher leaders who taught in the classroom represented in 
both the teacher and teacher leader results. Generally, we report the number and percentage 
of respondents who selected different multiple-choice options. In a few cases, where responses 
are numerical rather than categorical, we present means rather than proportions. Descriptive 
statistics for all multiple-choice items are presented in full in Appendix A for teachers, Appendix 
B for teacher leaders, and Appendix C for administrators. Responses are organized by domain 
and specific questions are identified by question number. For example, we summarize the 
domain of LDC participation in one section, but rely on information from four questions. We 
preface teacher questions with “T,” teacher leader with “TL,” and administrator questions with 
“A”. 

LDC CoreTools. The first stage of our analysis examined the proportion of all participants 
who created CoreTools user accounts, and engaged in three key behaviors: viewing, editing, 
and commenting on modules. We then analyzed each of the three key measures of 
participants’ interaction with the LDC online tools, and reported frequencies and/or descriptive 
statistics (e.g., minimum, maximum, mean, standard deviation) as measures of participants’ 
engagement with the online LDC system. In addition to reporting the overall results, we provide 
results by cohort, the content area taught (ELA, social studies/history, or science), participant 
role (teacher, teacher leader, and administrator), and school level (elementary, K-8, middle, 6— 
12, and high) whenever feasible. Finally, we explored the difference in CoreTools engagement 
between two groups of teachers: those that completed and taught modules and those who did 
not appear to complete the design and implementation process. 


Modules. We used descriptive statistics (means, standard deviations, and percentages) to 
analyze overall and subgroup performance for each content area and school level on each of 
the six dimensions. Additionally, generalizability theory (G theory) was used to examine 
potential sources of error during the rating process to help determine the validity of the scores 
as well as the construct validity of the rubrics (see Shavelson & Webb, 1991). Finally, teacher 
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comments during the debriefings were examined to determine other potential issues with the 
rubrics and/or rating process. 


Fidelity of implementation analysis. Fidelity of implementation analysis involved 
descriptive analyses of a variety of data sources. LDC and CRESST staff worked collaboratively to 
construct the fidelity matrix, with LDC staff setting the thresholds for adequate implementation 
on each indicator and at each level of implementation (teacher/module, school, and program). 
Although the process produces school level fidelity scores for all indicators, the initial unit of 
implementation was either module, teacher, or school depending on the indicator. 


Three indicators were based on data collected via CRESST’s teacher survey. Data on these 
selected survey items were used to produce school- and program-level fidelity scores. For one 
of the three survey-based indicators (perceived effectiveness of engagement in LDC on teacher 
competencies), we created an index variable by averaging across multiple items. Three 
indicators were based on data from the PLC Reflection Form collected by LDC, which was filled 
out by teacher leaders at the school level and used to capture information on both full PLC 
sessions and one-on-one planning and progress calls between the teacher leader and coach. 
Two indicators were based on LDC administrative records capturing attendance by teacher 
leaders and administrators at in-person meetings occurring four times per year. Finally, six 
indicators were based on LDC’s CoreTools analytic data capturing a variety of behaviors 
including teachers’ viewing of LDC online course content in the LEARN portion of CoreTools, 
editing of modules, and uploading of student work; and coaches’ commenting and providing 
peer review on modules. 


The fidelity matrix can be found in Appendix G. The matrix outlines a process whereby 
fidelity scores on each indicator are computed for each school, and for the program as a whole. 
The matrix also produces a score for whether the program met fidelity for each of the four 
components. As described in detail in the fidelity matrix in Appendix G, schools were rated on 
four-point scales (from 0 to 3) on each indicator. At the school level, adequate implementation 
was defined as a score of at least 2 (with the exception of principal mini-task observation where 
school-level fidelity was set at a score level of 1). For many of the indicators, the four-point 
scale was based on the proportion of teachers who met a certain implementation threshold. In 
order for the program as a whole to meet fidelity on a particular indicator, a certain proportion 
of schools had to meet the school-level threshold. For the indicators under Key Component 2, 
program-level fidelity was met if half or more of the schools met fidelity. For all of the other 
indicators (under Key Components 1, 3, and 4) program-level fidelity was met if three quarters 
of schools met fidelity. 


Table 2.6 summarizes the matrix indicators, and the criteria for meeting school and 
program level fidelity. 
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Table 2.6 


Summary of Fidelity Matrix Indicators and the Criteria for Meeting School and Program Level Fidelity 


% of Schools 
Meeting Fidelity 
Needed to Meet 


Criterion for Meeting Fidelity at School Fidelity at 

Key component Indicator Level Program Level 
Key Component Teacher attendance at weekly 75% of PLC teachers attended 80% of 75 
1: Common PLC meetings meetings 
Planning Time Amount of time spent on LDCin PLC sessions typically met for 60 minutes 75 
for LDC PLC sessions plus 
Professional 
Learning Exposure to LDC LEARN content 75% of PLC teachers viewed 60% of 75 
Community during 1st instructional cycle sessions in 1st LEARN instructional cycle 
with Exposure to LDC LEARN content 75% of PLC teachers viewed 60% of 75 
Synchronous during 2nd instructional cycle sessions in 2nd LEARN instructional cycle 
Coach Support Perceived effectiveness of Teacher survey respondents on average 75 

engagement in PLC onteacher —_ reported moderate improvement in key 

competencies competencies 
Key Component Coach comments on modules 75% of modules developed at the school 50 
2: and linked to LEARN courses received at 
Asynchronous least two comments from LDC coaches 
Support from Coach formative peer review on 75% of modules developed at school and 50 
LDC Coaches modules linked to LEARN courses received feedback 

from LDC coaches via peer review rubric 

Teacher perception of the 75% of teacher survey respondents 50 

helpfulness of coach written reported coach written feedback was at 

feedback on modules least moderately helpful 
Key Component Module editing 75% of PLC teachers edited task and either 75 
3: Teacher standards or text in at least 1 module 
Implementation Module implementation 75% of PLC teachers uploaded student 75 
Activities work to at least 2 modules 
Key Component Frequency of coach/teacher Teacher leaders had at least 9 planning / 75 
4: Leadership leader monthly meetings progress calls with coaches in school year 
suppert at Administrator attendance at School administrator participated in at 75 
Different Levels quarterly in-person least 2 out of 3 in-person meetings 

administrator meetings 

Teacher leader attendance at Teacher leader participated in at least 2 out 75 

quarterly in-person teacher of 3 in-person meeting 

leader meetings 

Principal mini-task observation 75% of teacher survey respondents 75 


reported having their LDC instruction 
observed by a school administrator 
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Student outcomes. We employed a quasi-experimental design to examine the effect of LDC on 
the New York State ELA assessment scores of students in the participating LDC elementary and 
middle schools in 2017-2018. Before conducting the analysis, we used a two-step matching 
process to identify a reduced pool of comparison students and teachers at schools with similar 
characteristics to the schools in the intervention sample. 


To accomplish this, we first identified the five most similar control schools for each 
intervention school based on a Euclidian distance measure, by using the nearest neighbor 
analysis option in SPSS 24.0 (see Fix & Hodges, 1951; Wang, Neskovic, & Cooper). The variables 
used in this process were the percentage of students eligible for free or reduced price lunch, 
the percentage of African American students, mean prior student achievement in ELA, mean 
prior student achievement in math, the average attendance rate of teachers, the percentage of 
teachers with three or fewer years of teaching experience, and the school grade span where 
feasible. Once the pool of comparison schools was identified, their students and teachers were 
also identified and student-level matching was conducted so that the resulting student sample 
would resemble the type of sample one would expect to obtain through random assignment. 


The student-level matching technique we employed was coarsened exact matching (CEM) 
(lacus, King & Porro, 2011). CEM is a flexible matching approach with many favorable 
properties, and allows the researcher to specify the precise conditions under which students 
are matched. For categorical variables, such as race/ethnicity or free or reduced price lunch 
status, this can entail exact matching, while for continuous measures, such as prior individual 
student achievement and aggregate class level achievement, cut-points for matching can be 
specified. With this approach we were able to set precise cut-points on the most important 
prior indicators, such as prior academic achievement, to ensure that where possible every 
treatment student was matched with a suitable comparison. Student matching variables we 
used in CEM included Hispanic, Black, poverty status, female, English language proficiency 
(English language learner), special education status, mean prior achievement in math and ELA, 
and grade level. 


During matching we also included a few variables capturing information on the teachers 
and peers to which students were exposed. These variables included mean prior ELA 
achievement of the student’s peers in her core content classes, and the average years of 
teaching experience of student’s core content teachers. 


The typical structure of course taking at the middle school level involves students 
potentially being exposed to multiple teachers, with each responsible for a different core 
content class. Specifically, middle school students in the study had exposure opportunities 
across three content areas (ELA, social studies/history, and science) taught by intervention 
teachers. As a result, students were not necessarily nested under individual teachers, but 
instead were likely to have connections to multiple teachers in the available time period prior 
to each testing outcome (students at the elementary school level were also sometimes exposed 
to multiple teachers but to a lesser extent). Therefore, LDC effects were estimated using an 
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extension of the standard multilevel modeling framework known as multiple membership 
multiple classification (MMMC) models (Browne, Goldstein, & Rasbash, 2001). 


These models can account for complex classification structures, such as the LDC context, 
in which students are nested within schools but are also members of multiple classes led by 
different teachers who may or may not be implementing LDC. MMMC has the flexibility to 
account for this type of complex nesting structure in which students are hierarchically nested 
under schools but may have one-to-many relationships with teachers. There are three 
classification levels in the models we employ: students, teachers, and schools. In the MMMC 
modeling approach, each observation at the lowest level represents one student. Figure 2.1 
shows how students can be exposed to teachers in different content areas, who may or may 
not be participating in LDC. Our MMMC modeling approach can account for this complicated 
structure. 


Teacher 1 (ELA) 


Teacher 2 (History) Teacher 3 (Science) 


LETNS< | 


Student 1 Student 2 Student 3 Student 4 Student 5 Student 6 


Figure 2.1. Variation in Student Exposure to Teachers 


In the MMMC modeling approach, each observation at the lowest level represents one 
student. The weight each teacher receives with respect to each student is dependent on the 
student’s exposure to his or her teachers in each of the three core content areas. The total 
student exposure adds to a unity (i.e., a possible total exposure of 1) across their courses in the 
three content areas in a given school year. While this general weighting approach applies to 
both elementary and middle schools, the course structure of the datasets required us to use 
different weighting procedures in elementary and middle school. 


In both the elementary and middle school datasets, students were linked to teachers 
through statewide course identifiers and accompanying beginning and ending dates specifying 
the time each student was enrolled in a given course under a specified teacher. Students could 
potentially have data records connected to multiple teachers covering varying time periods in 
the same course. For each of the three core content courses we then collapsed the links into a 
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single measure of the days of potential content exposure preceding the assessment period. The 
codes aligning with the three core content areas of interest for our study were based on the 
School Codes for Exchange of Data (SCED) code handbook. In elementary school, in the event 
that a student was exposed to more than one teacher, each content area was given equal 
weight in distributing teacher/student exposure. For example, if a student was enrolled for 
both ELA and social studies/history under one teacher, then that teacher was coded as .67 for 
having contributed to two thirds of the students’ core curriculum exposure. If the same student 
enrolled in science with a different teacher than the one who was linked to their course marks 
in ELA and social studies/history, then that science teacher would have been coded as .33 and 
all other teachers in the sample would have been coded as zero. This would then result in the 
student’s exposure adding to a unity (1). 


In middle school, students’ exposure to teachers at the course level in the three core 
content areas was coded in the same manner as in the elementary grades based on enrolled 
time preceding the assessment period. A difference in our middle school coding process was 
that we did not force each core content area into equal weighting. Instead each core content 
area exposure contributed to a core content area total sum which formed the basis from which 
the weights were proportioned. Most commonly a student had equivalent days of core 
instruction exposure in each the three content areas (often 215 days in each content area). In 
that scenario, if a student had exposure to three different teachers, then each teacher would 
contribute one third (.33) of the overall core curriculum exposure and all other teachers in the 
sample would be coded as zero. However, in addition to the typical core science course, extra 
core science courses were also included in the LDC analysis (for example a Grade 8 student 
taking biology), which made it possible then for a student to accumulate more units in science 
than in the other two content areas. The weighting in middle school was always distributed as a 
proportion of the total exposure days in the three content areas. Therefore, if a student 
accumulated 300 science days (across two courses), two hundred social studies days, and two 
hundred ELA days, the base number of instruction days would be 700 days. If, using that same 
scenario, the same teacher taught both the typical core and biology courses then that teacher 
would contribute three-sevenths (.43) of the overall core curriculum exposure with the social 
studies and science teachers contributing two-sevenths (.285) each, again resulting in the 
student’s exposure adding to a unity (1). Tabular versions of the above examples can be seen in 
Appendix G. 


For this study, we modeled the treatment intervention variable as a fixed effect at the 
student level in two different ways. The first dosage dependent approach takes into account 
the students’ level of exposure to the intervention teachers. In this approach, the treatment 
was structured as a continuous response variable, coded as zero for comparison students and 
coded as a positive value for treated students, albeit never exceeding one. The positive value 
assigned to treated students in the dosage-dependent approach was simply the sum of the 
intervention teacher weights linked to the treated student. The second approach was dosage 
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independent and classified any student exposed to an intervention teacher via at least one 
course as a treated individual. In this approach the treatment variable was dichotomous (coded 
as one for treated students and zero for comparison students). 


As with other multilevel models, MMMC models account for the non-independence of 
observations within cluster by adjusting the inferences on parameter estimates for the 
correlations between responses in a cluster. This modeling approach, however, becomes 
computationally cumbersome using traditional frequentist estimation methods. As 
recommended by Browne et al. (2001) we instead employed Bayesian methods using Monte 
Carlo Markov chain (MCMC) techniques to best address this issue. Multilevel models 
incorporate demographic and achievement variables used in the matching design as covariates, 
making the findings “double robust” (characteristics controlled for in both matching and 
outcomes analysis stages). Student demographic and prior achievement variables that were 
used in the matching process were also included as covariates in the MMMC model. In the 
elementary analysis, the variable for grade level did not meet our inclusion criteria (the p value 
was >0.20), and was not included in the final elementary models. Full model specifications can 
be found in Appendix G. 
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3.0 Survey Analysis 


In this chapter we present the survey results. First, we summarize teachers’ responses. 
We then summarize the teacher leaders’ responses, followed by the administrators’ 
responses.2 Whenever we felt a comment from an open-ended response might clarify, 
illustrate, or corroborate a finding, we included that comment in the appropriate section. 
Within each of these sections, we organize results by the following domains: LDC participation; 
professional learning community and teacher collaboration; LDC training and support; LDC 
module implementation; leadership support; impact; and issues of scale-up and sustainability 
and facilitators and barriers. The last section summarizes the results about program efficacy 
and improvement, which were completed by all three participant types. We end witha 
summary of results. 


As previously noted, we use acronyms to identify which participants answered specific 
questions for each domain. We preface teacher items with “T,” teacher leader with “TL,” and 
administrator questions with “A.” For example, LDC Participation (T1—4) indicates that teacher 
survey items 1-4 are used to provide information on LDC participation. Survey questions and 
descriptive results are presented in full in Appendix A for teachers, Appendix B for teacher 
leaders, and Appendix C for administrators. 


3.1 Teacher Survey Results 


A total of 231 teachers at 45 schools completed the survey. Ninety teachers (39%) taught 
in 14 elementary schools, 34 teachers (15%) taught in eight K-8 schools, 79 teachers (34%) 
taught in in 18 middle schools, and 28 teachers (12%) taught in 5 6-12 schools (see Table 3.1). 
By cohort, 35 teachers (15%) were returning teachers from last year (Cohort 1 returning), 47 
teachers (20%) were new teachers in Cohort 1 schools (Cohort 1 new), and 149 (65%) were new 
teachers from new schools (Cohort 2). 


In addition to producing descriptive statistics on the overall sample, we also analyzed 
results separately for elementary and secondary level teachers and by cohort. We highlight 
important differences between these subgroups, when they are apparent. Note however, that 
observed differences must be treated with caution, given the overlapping membership within 
the categories. That is, cohort differences are confounded with elementary versus secondary 
comparisons given that Cohort 2 schools were much more likely than Cohort 1 schools to be 
elementary. Cohort 1 versus Cohort 2 comparisons also are complicated by the presence of a 
substantial number of Cohort 1 new teachers, whose schools are in their second year of 
implementation but who personally are in their first year of implementation. Further these 
teachers were sometimes joining PLCs that started in 2016-2017 and sometimes in completely 
reconstituted PLCs in returning schools. 


As noted earlier, some of the teacher leaders were also classroom teachers, coaches, or coordinators. Because of 
this, some teacher leaders were represented in both the teacher leader and classroom teacher samples. 
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Table 3.1 
Schools and Teachers Completing the Survey in 2017-2018 


N of Cohort NofCohort N of Cohort Total 


N of 1 returning 1 new 2teachers Teachers 
School type schools teachers teachers 
Elementary schools 14 8 4 78 90 
K-8 schools 8 6 4 24 34 
Middle schools 18 15 30 34 79 
6-12 schools 5 6 9 13 28 
Total 45 35 47 149 231 


Secondary teachers reported teaching 1 to 20 classes (mean = 4.6) and using LDC 
materials in 0 to 13 classes (mean = 2.9). Forty-four percent reported using LDC in ELA; 40% in 
history/social studies; 21% in science; 12% in other areas (arts, languages, P.E.); and 1% in 
mathematics (see Table 3.1). 


Teachers' Content Area Implementations 


50% 
45% 


40% 
40% 
35% 
30% 
9 

25% 21% 
20% 
15% 12% 
10% 

= Zz = 

0% —_ = 

ELA 


History/Social Studies Science Other Math 
(n = 51) (n = 47) (n = 24) (n = 14) (n =1) 


44% 


Figure 3.1. Teachers’ content area implementations. 


LDC participation (T1—4). Over three quarters of responding teachers (78%) reported that 
2017-2018 was their first experience with LDC. The remaining 52 teachers (23%), reported that 
they had previously used LDC tools and averaged teaching over two modules in the prior year 
(mean = 2.4), with a wide range of 0 to 5 modules. These teachers also reported teaching over 
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three mini tasks outside of the modules in the prior year (mean = 3.6), with a range of 0 to 20 
mini-tasks. 


Professional learning community and teacher collaboration (T5—9, T31, T38). Almost all 
teachers (92%) participated in a PLC focused at least partly on implementing LDC. Of the 17 
teachers who had not participated in a PLC, two teachers reported that they did not use any 
LDC tools in their planning or instruction, while 15 teachers (88%) reported they had used LDC 
tools in their planning or instruction. Of these 15 teachers, 11 used CoreTools to access 
modules; nine used CoreTools to design modules; six were given modules by other teachers; 
and four took online LDC courses. We hypothesize that these 15 teachers, given their 
engagement, may have partially participated in the PLC, perhaps by initially attending meetings 
and later dropping out. 


Half of the teachers (51%) met with their PLCs once a week. Nearly a third (30%) met 
every other week, and a small percentage (15%) reported meeting monthly. Seven teachers 
(4%) reported meeting twice a week or more often, and one teacher (1%) reported meeting less 
than once a month. The most common reason cited for not meeting weekly was that PLC 
members had other priorities (61%). Other, less frequently cited reasons were lack of protected 
PLC time (14%); not a priority of the administration (7%); lack of participating teachers (4%); 
limited interest in attending meetings (2%); and not enough support from the teacher leader 
(1%). 


The majority of teachers reported that LDC PLC meetings lasted 45 minutes or more: 58% 
reported meetings of 45 minutes to an hour (with 71% of elementary teachers meeting for this 
amount of time, compared to 47% of secondary teachers), and 22% reported that meetings 
lasted longer than an hour. Twenty percent reported meetings of less than 45 minutes. A 
majority of teachers (74%) agreed or strongly agreed that their PLCs were given sufficient time 
to meet. However, in the open-ended responses, 65 teachers (36% of 179 teacher commenters) 
indicated that time was a barrier to effective implementation. These comments included time 
issues, such as regularity of meetings and the need for teachers to have time outside of regular 
formal meetings to discuss and implement LDC, as well as suggestions for training to start 
earlier. 


Beyond formal PLC meetings, half of teachers (50%) said they had informal discussions 
about LDC with their colleagues once a week or more; 18% every other week; 18% once a 
month; and 14% less than once a month. 


Teachers also were positive about the effects of their PLC participation on their 
collaboration with other teachers. A majority of teachers (84%) agreed or strongly agreed that 
after participating in LDC they were more likely to collaborate with other teachers on designing 
instruction. Over three quarters (79%) reported that participating in LDC helped them develop 
working relationships with teachers in different grades or subjects. Sixty percent reported 
sharing their LDC work with colleagues who were not in LDC. 
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LDC Training and Support (T10—T13). Teachers evaluated the three primary types of 
training and support offered by LDC: PLC time, online course materials, and virtual coach 
support during and outside of PLCs. 


Teachers found PLCs to be moderately or very effective in the following ways: creating an 
environment in which teachers were comfortable working with each other (94%); fostering an 
environment where teachers shared instructional plans with colleagues (92%); allowing space 
for shared problem solving (90%); and helping teachers learn to develop modules (89%) (see 
Figure 3.2). Cohort 1 new teachers were slightly less likely to report that their PLC was effective. 


Effectiveness of Teacher's PLC 
100% 94% 92% aa ee 
90% 
80% 
70% 
60% 
50% 
40% 
30% 


20% 11% 
0, 0 
ae 6 10% 


0% 


Creating an environment Fostering an environment Allowing space to share Helping teachers to 
in which teachers were _ where teachers shared student work improve their LDC 
comfortable working — their instructional plans (n = 213) instructional plans 
together with colleagues (n = 213) 
(n = 213) (n = 213) 
@ Not effective/A little effective @ Moderately effective/Very effective 


Figure 3.2. The effectiveness of teacher’s PLC (T10). 


The majority of teachers also reported satisfaction with different aspects of the online 
course materials by rating them good or excellent, including: relevance of information (84%); 
opportunity to extend learning (83%); clarity of information (81%); usefulness of resource 
documents such as the LDC Pitfall Checklist (78%); helpfulness in creating modules (78%); 
usefulness of videos (72%); and ease of use (70%) (see Table 3.2). Cohort 2 teachers were 
overall less positive about the online course materials with good or excellent ratings ranging 
from 63% to 82% depending on the item, compared to the Cohort 1 teachers whose ratings 
ranged between 77% and 95%. 
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Table 3.2 
Rating Online Course Materials (n = 213) 


Poor Fair Good Excellent 


How would you rate each of the following aspects of the online course material (in the Learn tab in LDC 
CoreTools) that your coach used or directed you to use? 


Clarity of information presented 3% 16% 50% 31% 
Relevance of information presented 1% 15% 55% 29% 
Ease of use 8% 23% 47% 23% 
ie a documents (e.g., LDC Pitfall Checklist, CCSS 3% 18% 8% 30% 
Usefulness of videos 6% 22% 52% 21% 
ane tae material helped me to create and/or A% 19% 51% 27% 
Opportunity to extend learning when needed or desired 4% 13% 52% 31% 


Almost all teachers (99%) felt they were able to get the feedback and support they 
needed from their LDC coach and reported that coaches provided written feedback in a timely 
manner. 


In addition to direct support during PLCs, at least half the teachers found the following 
types of asynchronous coach support found to be moderately or very helpful: written feedback 
in CoreTools (79%; 8% did not use); individual Zoom conference (77%; 11% did not use); and 
email or phone communication (77%; 13% did not use). In general, Cohort 1 returning teachers 
found the asynchronous coach supports more helpful than the other two teacher groups, 
especially with regard to written feedback: 91% of Cohort 1 returning teachers found it to be 
moderately or very helpful, compared to 76% of Cohort 2 teachers and 82% of Cohort 1 new 
teachers. 


LDC implementation (T14-29). This domain covers questions on module creation (T14- 
17), module peer review (T27-29), and classroom implementation (T18-26). 


Module Creation (T14-17) and Module Peer Review (T27-29). Teachers adapted or 
created two types of LDC instructional products: mini-tasks are short, generally taking one 
period, and focus on a specific skill; modules are longer, more complex units comprised of a 
series of mini-tasks, which build to a culminating “teaching task,” an end of module writing 
assignment. 


Using existing LDC templates and exemplars, teachers individually or collaboratively 
adapted between 0 and 10 modules during the year. Among the 184 teachers who answered 
this question, the majority of teachers (68%) adapted at least two modules, with 69 teachers 
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(38%) adapting two, 39 teachers (21%) adapting three, and 16 teachers (9%) adapting four or 
more modules. In contrast, 17 teachers (9%) adapted no modules and 43 teachers (23%) 
adapted one. The mean number of modules teachers adapted was 2.1. 


Creating modules from scratch, teachers individually or collaboratively constructed from 0 
to 20 modules (mean = 1.4). The majority (69 teachers or 38%) constructed one new module, 
while 53 teachers (29%) created zero new modules, 42 teachers (23%) created two, 13 teachers 
(7%) created three, and seven teachers (3%) reported creating four or more. 


Teachers also reported how they constructed modules. A majority of teachers (62%) 
worked in teams of two or more to create modules. About half (48%) created modules 
individually, with secondary teachers (57%) doing this more frequently than elementary 
teachers (37%). A relative few (18%) collaborated with the entire PLC to write modules. 


A little over a third of teachers (35%) reported attending a Peer Review/Curriculum 
Alignment Workshop during the 2017-2018 school year. Similarly, a little over a third of 
teachers (38%) submitted their modules for LDC National Peer Review, and this small 
percentage was to be expected given the preponderance of teachers were new to LDC this 
year. Of the 79 teachers who reported submitting modules for peer review, 29 (37%) found the 
process very helpful, 19 (24%) moderately helpful, 23 (29%) a little helpful, and eight (10%) not 
helpful. Cohort 1 new (67%), Cohort 2 (60%), and secondary (68%) teachers were more likely to 
report that the National Peer Review was helpful than Cohort 1 returning (50%) or elementary 
(53%) teachers. 


Reflecting on the instructional strengths of their modules, most teachers felt confident in 
their ability to execute all nine of the instructional features of LDC modules that were tracked: 
teachers reported being able to select focus standards for the module culminating writing 
assignment (92% felt they were able to accomplish this to a moderate or great extent); create a 
standards-driven writing assignment (90%); identify the skills needed to complete the 
assignment (89%); create an assignment relevant and engaging to students (88%); select high 
quality texts (85%); plan multiple methods to assess progress (85%); assess the quality of 
writing assignments and instructional plans (82%); create daily skills lessons (79%); and 
differentiate instruction (76%). As the data in Table 3.3 show, teachers expressed the highest 
confidence in their ability to select focus standards and to create a standards-driven writing 
assignment. 
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Table 3.3 
Rating on Teacher Skills Associated with Creating LDC Modules (n = 211) 


Not at A little Amoderate Agreat 
all bit extent extent 


Please indicate to what extent you were able to do each of the following when creating LDC modules. 


Select focus standards for a writing assignment 1% 7% 36% 57% 
Create a standards-driven writing assignment 1% 10% 36% 54% 
Select high quality, complex texts and other materials that allowed 2% 13% 39% 46% 
students to engage in deeper learning 
Identify the skills students need to develop to complete a writing 1% 10% 42% 47% 
assignment 
Create daily lessons to teach the skills a student needs to complete a 5% 16% 42% 37% 
writing assignment 
Differentiate instruction by incorporating multiple ways of thinking, 6% 18% 42% 35% 
various levels of complexity, and multiple modalities 
Plan for a variety of methods to assess student progress (e.g., rubrics 3% 13% 48% 36% 
and/or mini-task scoring guides) 
Assess the quality of writing assignments and/or instructional plans 2% 16% 44% 38% 
using Peer Review/Curriculum Alignment Rubric (e.g. Task Pitfalls 
Checklist, rubric indicators) 

3% 10% 43% 45% 


Make a writing assignment relevant and engaging for students 


There were some differences by cohort in these ratings of LDC instructional features. 
Cohort 1 returning teachers felt more positive in their ability to execute LDC instructional 
features than the other two teacher groups. For example, Cohort 1 returning teachers 
appeared confident in their ability to select high quality texts (100%), create daily skills lessons 
(94%), or differentiate instruction (88%). 


Classroom Implementation (T18-26). Toward the beginning of the year, teachers were 
asked to “Find and Teach” a module from the CoreTools library. The majority of teachers (62%) 
reported that they did “Find and Teach” a module from CoreTools, with more elementary 
teachers (72%) doing so than secondary teachers (53%). Of those teachers, 77% reported 
making adjustments to the modules they found to adapt them to their classroom needs. 


After implementing the “Find and Teach” module, teachers were also asked to adapt, 
refine, and/or develop a module. The majority of teachers adapted and refined an existing 
module in the LDC Library in CoreTools (126 teachers, 60%) while the remainder (84 teachers, 
40%) created a module from a template in CoreTools. A greater proportion of elementary 
teachers (72%) reported adapting an existing module compared than secondary teachers (50%). 
Most teachers (88%) reported teaching these modules by the time they completed this survey, 
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with nine teachers (4%) reporting that they planned to teach the module before the end of the 
2017-2018 school year, 13 teachers (6%) planning to teach it during the next school year, and 
three teachers (1%) stating that they did not have any plans to teach the module. 


After creating or adapting modules, teachers implemented them in their classrooms and 
reflected on their success. Teachers reported implementing from 0 to 20 modules (mean = 2.5), 
as well as from 0 to 50 mini-tasks (mean = 3.7) that were not part of modules (see Table 3.4 for 
more information). Most teachers (173 out of 211, or 82%) reported implementing at least the 
recommended two modules over the course of the school year. The majority of Cohort 1 
returning teachers reported going beyond the recommended two modules and teaching three 
or more modules (69%). 


Teachers reflected on various dimensions of their module implementation and the 
majority were positive about all aspects. Based on teacher reports, 88% were able to find 
evidence of student performance on targeted standards in student work to a moderate or great 
extent; 87% engaged students in understanding the assignment and the rubric for evaluating 
student work; 86% felt that they engaged students in accessing complex text; 84% felt they 
provided feedback to students using assignment rubrics; 83% used evidence of learning to 
modify instruction; and 81% systematically collected information about student learning. In 
addition, across all six dimensions, between 37% and 44% of the teachers reported that they 
were able to accomplish the activity “to a great extent,” which indicated that about a third of 
more of the teachers felt very confident about their implementation. 


Cohort 1 returning teachers were overall more positive than their peers about the above 
aspects of module implementation. In particular, Cohort 1 returning teachers engaged students 
in accessing complex text (98%), found evidence of student performance on targeted standards 
(98%), engaged students in understanding the rubric (97%), and provided feedback (97%). 
Elementary teachers reported (93%) engaging students in accessing complex text more than 
secondary teachers (72%). 
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Table 3.4 
Number of Modules and Mini-Tasks Implemented (n = 211) 


Numbers of Modules or Mini- 


jacks implemented in'2017: N of Vege ey one N of Ve ee 
2018 
0 9 42 
1 29 27 
2 98 31 
3 52 36 
4 11 18 
5 5 17 
6 3 12 
7 0 2 
8 1 5 
9 0 1 
10 0 10 
11 0 1 
12 1 3 
15 0 3 
16 1 0 
20 1 2 
50 0 1 


Leadership support (T33-37). This domain covers questions on teacher leader support 
and administrator support (T33-36) and teacher leadership role in LDC (T37). 


Teacher Leader Support (T33). Teacher leaders were school staff who provided logistical 
support to the PLCs. Responses indicated that the vast majority of teachers were very satisfied 
with the level of teacher leader support. The teacher leaders were approachable (99% agreed 
or strongly agreed); effectively supported the PLC meetings (96%); effectively invited teachers 
to join LDC (96%); helped teachers align LDC to broader school goals (93%); and gave useful 
feedback (92%). 


School Administrator Support (T34-36). School administrators were principals, assistant 
principals, or other instructional leaders who observed teachers in action and provided other 
leadership, such as protecting time for PLCs to meet. Teachers provided feedback about the 
support they received from their school administrator. The involvement of school 
administrators, as reported by teachers, was varied. 
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A little under half of teachers (44%) reported that their administrator attended less than 
one quarter of the LDC PLCs, and almost a quarter (22%) said their administrator attended 
about one quarter of the PLCs. Smaller proportions of teachers reported their administrator 
attended PLCs about half the time (14%), about three quarters of the time (5%), and more than 
three quarters of the time (15%). Cohort 1 returning teachers reported the least administrator 
participation compared to the other teacher groups. 


Forty percent of teachers (40%) reported never being observed by an administrator while 
teaching an LDC mini-task, with a lower percentage of Cohort 1 returning teachers (28%) 
reporting no administrator observations compared to the other teacher groups (41-47%). 
About a third (32%) reported being observed once by their administrator; 18% were observed 
three or more times; and the remaining 11% were observed twice. 


Almost all teachers (88%) agreed or strongly agreed that their administrator encouraged 
LDC participation. According to teachers, administrators were able to allocate resources to 
ensure PLCs had time to meet (81%); made formative assessment a priority (81%); had a firm 
understanding of LDC (80%); and communicated how LDC supported school initiatives and goals 
(79%). Most administrators also reportedly used LDC to implement standards-driven 
assignments within existing curriculum (71%) and provided feedback about LDC planning and 
instruction (62%). About half (48%) disagreed that administrators expressed concern that LDC 
was taking time away from other instructional priorities. 


In general, Cohort 1 returning teachers were more likely to rate their administrators as 
supportive or effective than did the other two teacher groups. Cohort 1 returning teachers 
(91%) reported that their administrator allocated resources to ensure that the LDC team could 
meet, compared to their school peers newly joining LDC in 2017-2018 (74%). More Cohort 1 
returning teachers (94%) reported that their administrators made formative assessment a 
priority than teachers in the other groups (both at 79%). More Cohort 1 returning teachers 
(88%) felt that their administrators used LDC to implement standards-driven assignments 
within the existing curriculum than Cohort 2 teachers (66%). 


Teacher Leadership Role (T37). Many teachers who were not playing a formal teacher 
leader role as part of the LDC program nevertheless felt that LDC allowed them to exercise 
leadership in their schools. Most teachers agreed or strongly agreed that LDC gave them the 
opportunity to shape LDC implementation with the teacher leader and administrator (63%) and 
helped them set instructional goals for LDC work at their schools (61%). Two-thirds (66%) 
expressed interest in learning more about facilitating LDC implementation at their schools. Over 
half of teachers (55%) felt they were able to affect the direction of LDC at their site by having 
the opportunity to discuss expansion of LDC implementation at their school in future years. 


Cohort 1 returning teachers were overwhelmingly more positive about their involvement 
in LDC decision making compared to the other two teacher groups. Eighty-four percent of 
Cohort 1 returning teachers agreed or strongly agreed that they were involved in setting 
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instructional goals for LDC in their schools compared to 64% of Cohort 1 new and 56% of Cohort 
2 teachers. Similarly, 78% of Cohort 1 returning teachers reported having discussed how to 
expand LDC at their schools in future years and had the opportunity to work with their LDC 
teacher leaders and administrators to shape LDC implementation, compared to smaller 
proportions of Cohort 1 new teachers (50% and 58%, respectively) and Cohort 2 teachers (52% 
and 61%, respectively). Most Cohort 1 returning teachers (87%) reported that they were 
interested in learning more about how to lead LDC implementation at their schools (compared 
to 72% Cohort 1 new teachers and 60% Cohort 2 teachers). 


LDC Impact (T30-32). This domain covers questions on LDC impact on teacher practice 
and learning (T30-31) and student learning (T32). 


Impact on Teacher Practice and Learning (T30-31). We asked teachers about how LDC 
had changed their ability to practice key teacher skills. Specifically, we asked them to focus on 
the change from the beginning to the end of the current school year’s work with LDC. We also 
asked them to provide additional information about the impact of LDC on their instructional 
practice. 


Across all seven items asking about LDC-related pedagogical changes during the year, the 
majority of teachers rated themselves as having improved moderately or a great deal. 
Specifically, teachers perceived improvement in instructional planning by selecting focus 
standards (85%); creating standards-driven assignments (81%); identifying skills students 
needed to complete an assignment (81%); using evidence of student learning to modify 
instruction (80%); identifying patterns of student understandings and misconceptions (80%); 
creating daily lessons to teach the skills students needed (74%); and collecting information on 
students’ progress (73%). 


There were differences by cohort, with Cohort 2 teachers least likely to report positive 
change. Cohort 2 teachers ranged between 70% and 80% agreeing or strongly agreeing with 
statements on improving their skills, compared to 78-95% in the other two teacher groups. 
Seventy-six percent (76%) of Cohort 2 teachers agreed or strongly agreed that they improved 
on creating standards-driven writing assignments (compared to 94% of Cohort 1 returning 
teachers); 76% felt they identified skills students needed to complete an assignment (compared 
to 95% of Cohort 1 new teachers); and 74% thought that participating in LDC raised their 
expectations for students’ writing (compared to 94% of Cohort 1 returning teachers). 


Figure 3.3 reports data on other areas of LDC impact on teacher practice. The areas of 
widest reported LDC impact on teacher practice included helping teachers: collaborate with 
other teachers (84% agreed or strongly agreed); incorporate writing assignments into their 
classroom curriculum (82%); raise expectations for student writing (81%); develop relationships 
with teachers outside grade or subject (79%); incorporate state standards in classroom 
instruction (74%); and establishment of LDC modules as a part of existing instructional practice 
(71%). LDC had a slightly narrower but still apparent impact in the following areas: sharing LDC 
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work with colleagues outside of the LDC PLC (61%) and improving teacher evaluation ratings 
(60%). 


Fewer Cohort 2 teachers agreed or strongly agreed with all of the above statements 
compared to the other two teacher groups. Cohort 2 teachers ranged from 51% to 81% on 
agreeing or strongly agreeing with those statements, compared to 74-100% of new and 
returning Cohort 1 teachers. Seventy-three percent of Cohort 2 teachers agreed or strongly 
agreed that participating in LDC raised my expectations for students’ writing (compared to 97% 
of Cohort 1 new teachers); 62% of Cohort 2 felt using LDC modules became an important part 
of their teaching practice (compared to 90% of Cohort 1 new teachers); 69% of Cohort 2 felt 
that LDC helped them incorporate state standards in their instruction (compared to 84% of 
both Cohort 1 teachers); 76% of Cohort 2 said that LDC helped them incorporate writing 
assignments (compared to 95% of Cohort 1 new teachers); 51% of Cohort 2 agreed that LDC 
helped improve teacher evaluations (compared to 81% of Cohort 1 returning teachers); 72% of 
Cohort 2 said that participating in LDC helped developed working relationships with different 
teachers (compared to 100% of Cohort 1 returning teachers); and 54% of Cohort 2 teachers 
shared LDC work with other colleagues (compared to 75% of Cohort 1 returning teachers). 
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Impact on Teacher Practice and Learning 
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Figure 3.3. Impact on teacher practice and learning (T31). 


Impact on Student Learning (T32). 


Teachers were asked to evaluate the effect of LDC on student learning (see Table 3.5). On 
average, about three quarters of teachers agreed or strongly agreed that LDC impacted student 
learning in the following ways: increased content knowledge (79%); supported students to 
complete writing assignments (77%); increased student capacity to analyze components of the 
writing task (76%); improved quality of writing skills (75%); improved overall literacy 
performance (75%); increased performance on assessments throughout the year (74%); 
developed reading skills (73%); developed speaking and listening skills (68%); and developed 
skills needed for college and career readiness (67%). 


Table 3.5 


Impact on Student Learning (n = 210) 


Not at all A little Moderately A great deal 


Please indicate to what extent LDC had a positive effect on students in the following areas. 


Reading skills 9% 19% 52% 21% 
Content knowledge 5% 16% 52% 27% 
Ability to complete writing assignments 5% 18% 46% 31% 
Quality of students’ writing 6% 20% 48% 27% 
College and career readiness 11% 22% 48% 19% 
Capacity to analyze and understand the 6% 18% 48% 28% 
components of a writing assignment 

Speaking and listening skills 11% 22% 51% 17% 
Overall literacy performance 5% 21% 51% 24% 
Performance on assessments throughout the 7% 19% 53% 21% 
school year 


Cohort differences (15% or above) emerged for all but one of the statements on student 
learning, with Cohort 1 new teachers again appearing as the least positive and Cohort 1 
returning teachers (those who had actually had two years of LDC experience) generally the 
highest. Cohort differences to note for the following effects on student learning (moderately to 
a great deal) include: reading skills (67% Cohort 2, 91% Cohort 1 returning); ability to complete 
writing assignments (70% Cohort 2, 97% Cohort 1 returning); quality of students’ writing (70% 
Cohort 2, 94% Cohort 1 returning,); college and career ready skills (63% Cohort 2, 91% Cohort 1 
returning); analyze and understand writing assignment components (71% Cohort 2; 97% Cohort 
1 returning); speaking and listening skills (63% Cohort 2, 91% Cohort 1 returning); overall 
literacy performance (70% Cohort 2, 88% Cohort 1 returning); and performance on assessments 
throughout the year (68% Cohort 2, 94% Cohort 1 returning). 


Facilitators and barriers (T38). Successful implementation of LDC depends on a number 
of factors. We asked teachers to weigh in on the effect of these factors on implementation (see 
Figure 3.4). Most teachers (83%) agreed or strongly agreed that it was easy to find and adapt 
LDC mini-tasks. Similarly, most teachers (83%) thought that their school had adequate 
technology to support teachers’ use of LDC, although in open-ended responses, some teachers 
expressed concern about lack of technological knowledge which affected their access to 
CoreTools and the Zoom meetings. About three-fourths of teachers felt adequately prepared to 
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implement modules in the classroom (76%) and felt their PLC was given sufficient time to meet 
(74%), although many teachers cited time as a barrier in the open-ended responses. However, 
most teachers (60%) found it challenging to find content-rich texts to use with LDC modules. 


Factilitators and Barriers to Implementation 
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Figure 3.4. Facilitators and barriers to implementation (T38). 


There were a couple differences by cohort in how teachers responded to facilitators and 
barriers in LDC implementation. While all Cohort 1 returning teachers (100%) agreed or strongly 
agreed that it was easy to find and adapt LDC mini-tasks, 78% of Cohort 2 and 84% of Cohort 1 
new teachers felt the same way. Likewise, 94% of Cohort 1 returning teachers felt adequately 
prepared to effectively implement LDC modules in their classrooms, compared to 71% of 
Cohort 2 and 76% of Cohort 1 new teachers. 


3.2 Teacher Leader Survey Results 


This section presents the survey results from teacher leaders. At each school, the teacher 
leader coordinated PLC meetings with LDC coaches. Teacher leaders were either classroom 
teachers or coaches/coordinators. A total of 43 teacher leaders responded to the survey. 


LDC Participation (TL1-2). The majority of the 43 teacher leaders who responded to the 
survey (26, 61%) reported that they were new to LDC and did not have experience with LDC 
prior to the 2017-2018 school year. Of the 17 teacher leaders (40%) who had prior experience 
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with LDC, they reported teaching between 0 and 5 (mean = 1.9) LDC modules and between 0 
and 9 (mean = 2.0) LDC mini-tasks outside of the modules. 


Professional Learning Community and Teacher Collaboration (TL3-6). Almost one half 
(44%) of teacher leaders reported that that their PLCs met once a week or more frequently. 
Forty-two percent of teacher leaders reported that their PLCs met every other week. A few 
teacher leaders (14%) reported that their PLCs met once a month, and no teacher leaders 
reported that PLCs met less than once a month. The 24 teacher leaders whose PLCs did not 
meet weekly reported that the primary barriers were that PLC members had other competing 
priorities (71%,), PLC time was not protected (25%), and PLC members had limited interest in 
attending meetings (21%). 


Also similar to teacher responses, the majority of teacher leaders (56%) said these 
meetings lasted 45 to 59 minutes. Twenty-one percent (21%) reported that meetings lasted 60 
to 74 minutes, and 19% reported that meetings were less than 45 minutes. A small minority 
(5%) said meetings lasted 75 minutes or more. Forty-four percent of teacher leaders reported 
having informal discussions about LDC once a week, 19% said every other week, 16% reported 
twice a week or more, 14% said once a month, and 7% reported less than once a month. 


LDC Training and Support (TL7-12). Teacher leaders evaluated the effectiveness of the 
various training and support methods provided during LDC implementation. These included PLC 
meetings, online course material, and coach support. Their responses are slightly more positive 
compared to teacher ratings. 


Teacher leaders reported that the PLC meetings were effective. Most teacher leaders 
(91%) felt that the LDC PLC was moderately to very effective in fostering an environment where 
teachers shared instructional plans with colleagues. Teacher leaders also reported that their 
PLC was effective in each of the following ways: allowing a space to share student work (86%); 
creating an environment in which teachers were comfortable working with each other (83%); 
and helping teachers learn to improve their LDC instructional plans (79%). 


Online course materials used by the coaches during PLC meetings also received slightly 
higher ratings than overall teacher ratings. Most teacher leaders (84%) rated relevance of 
information and opportunity to extend learning when needed or desired as good or excellent; 
81% for helpfulness in creating modules; 79% for usefulness of resource documents; 79% for 
usefulness of videos; 77% for ease of use; and 71% for clarity of information. 


LDC coaches also were highly regarded by teacher leaders. All (100%) reported that they 
were able to get the feedback and support they needed from LDC coaches and that the coaches 
provided written feedback on modules in a timely manner. Different types of asynchronous 
coach support were rated by the majority of teacher leaders as moderately or very helpful. 
Teacher leaders generally found the support, when used, helpful. These supports were: email 
or phone communication (88% helpful; 5% did not use); individual video conferencing with 
coaches (88% helpful; 7% did not use); and written feedback in CoreTools (84% helpful; 5% did 
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not use). All teacher leaders (98%) reported that their LDC coaches connected them with 
additional resources when needed. Almost all teacher leaders (100%) agreed or strongly agreed 
that they were able to reach their coach with questions; 98% said that their coach responded 
quickly; 98% thought their coach was easy to work with; and 95% said the coach was 
knowledgeable and provided high quality guidance. Additionally, only 37% of teacher leaders 
reported that it was challenging to structure PLC time with the coaches. 


Teacher leaders had the opportunity to attend LDC professional development meetings, 
both in person and online (these were in addition to PLC meetings). The average number of 
meetings attended was 4.0, with a range of 0 to 15. With respect to LDC support outside of 
coaches, all teacher leaders (100%) agreed that LDC offered sufficient professional 
development opportunities for them to lead the initiative at their schools, and 95% agreed that 
there was adequate technical support for CoreTools’ issues. 


Module Creation (TL13-16). Teacher leaders reported the ways in which modules were 
created in their PLCs and judged the extent to which they were able to embed targeted 
instructional practices while creating the modules. 


Teacher leaders were asked how many existing LDC modules were adapted by their PLCs. 
Answers ranged from 0 to 15, with a mean of 3.2. Teacher leaders were also asked how many 
modules their PLCs created from scratch, either individually or as a group, and they reported 0- 
7 modules, with a mean of 2.0. Modules were most commonly created via a collaborative 
process, either by teams of two or more teachers (72%) or by the PLC as a whole (21%). Over 
half of the teacher leaders (51%) reported that modules were created by individual PLC 
members as well. 


In general, teacher leaders were extremely confident in their ability to carry out targeted 
instructional practices while creating their modules; their responses were slightly more positive 
than those of general teacher participants. Almost all teacher leaders (93%) felt they were able, 
to a moderate or great extent, create a standards-driven writing assignment; 93% made writing 
assignments relevant and engaging to students; 93% identified the skills students needed to 
complete the writing assignment; 93% assessed quality of writing assignments or instructional 
plans; 91% selected focus standards for the writing assignment; 91% were able to select high 
quality texts and other materials; 86% created daily lessons to teach the necessary skills; 84% 
differentiated and provided multiple opportunities for students to engage with the assignment; 
and 84% planned for a variety of ways to assess student progress. 


Impact on Student Learning (TL17). Teacher leaders were asked to evaluate the effect of 
LDC on student learning and here again their responses tend to be more positive than those of 
teacher participants. Teacher leaders agreed or strongly agreed that LDC impacted student 
learning in the following ways: improved quality of writing skills (86%); supported students to 
complete writing assignments (86%); increased performance on assessments throughout the 
year (84%); improved overall literacy performance (84%); increased student capacity to analyze 
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components of the writing task (81%); developed skills needed for college and career readiness 
(79%); developed reading skills (77%); increased content knowledge (77%); and developed 
speaking and listening skills (72%). 


Alignment (TL21). Teacher leaders were asked how LDC aligned with other instructional 
priorities and programs at their schools. Alignment was generally perceived to be quite high as 
reported in Figure 3.5. Eighty-one percent agreed or strongly agreed that LDC helped teachers 
create writing assignments to use within their existing curricula. The majority of teacher leaders 
(91%) viewed LDC as complementing other initiatives at the school; 88% thought LDC was a 
strategy for implementing statewide college and career-ready standards; 84% said LDC helped 
prepare students for state assessments; and 77% reported their school connected LDC to 
specific school goals. 


Notably, however, over three quarters (76%) believed that it was difficult for teachers to 
focus on LDC because of other competing priorities at the school, and 69% reported that time 
spent on LDC interfered with other initiatives. 


LDC Alignment 


Our school connected LDC implementation to our specific 63% 
schoolwide goals. (n = 43) 37% 
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Figure 3.5. LDC alignment (TL21). 
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Teacher Leader Role (TL20). Teacher leaders reflected on their leadership role in LDC 
implementation at their school sites. Overall, results were very positive and indicated that the 
majority of teacher leaders felt engaged by and invested in LDC implementation at their school. 
Almost all (91%) agreed or strongly agreed that they met regularly with their LDC coach to 
manage the work plan; 91% felt they were able to build their own capacity as an instructional 
leader as a result of their LDC leadership role; 88% were involved in adjusting LDC problems of 
practice at their school; 81% were involved in planning how to meet teacher learning needs by 
differentiating LDC implementation; 79% were involved in discussions about how to expand 
LDC implementation in future years; 79% believed their role as LDC teacher leader helped them 
advocate for additional resources on their campus; 70% were confident that they could lead 
their PLC in the future without the assistance of an LDC coach; and 63% met regularly with their 
school administrator. 


Leadership Support (TL18-19, T23). This domain covers questions on school administrator 
support (TL18-19) and district support (TL23). 


School Administrator Support (TL18-19). Similar to teachers’ responses, teacher leaders 
reported on school administrators’ involvement with LDC. Forty-four percent of teacher leaders 
(44%) reported their administrators attended less than a quarter of PLC meetings; 21% about a 
quarter; 23% about half; and 12% more than three quarters. 


A large majority of teacher leaders (93%) reported their administrators made formative 
assessment a priority at their school; 91% reported their administrators encouraged teachers to 
participate; 86% said their administrator communicated how LDC supported other school goals 
and initiatives; 81% said their administrators had a firm understanding of LDC; 81% reported 
that resources were allocated to ensure that the LDC team could meet; 81% said administrators 
used LDC to implement assignments with existing curriculum; and 68% said they received 
feedback about their LDC planning or instruction from their administrator. Notably, 58% of 
teacher leaders reported that their administrators expressed concern that LDC was taking time 
away from other instructional priorities, which is a higher percentage than that reported by 
teacher participants (48%). 


District Support (TL23). Almost half of teacher leaders (49%) agreed or strongly agreed 
that district professional development efforts aligned with LDC; 47% said that district leaders 
supported implementation of LDC; 42% thought district leaders had a firm understanding of 
LDC; 40% reported that district leaders visited their school to discuss LDC implementation; and 
35% agreed that district leaders were interested in implementing LDC at other schools. 


Scale-up and Sustainability (TL22). Regarding the future of LDC implementation, most 
teacher leaders (84%) agreed or strongly agreed that they expected most teachers to continue 
with LDC the following year; 74% felt teachers and administrators were committed to 
sustaining the initiative; 70% expected their LDC PLC to increase in size the following year; and 
70% reported that new collaborations were launched as a result of LDC (see Figure 3.6). Most 
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did not think that non-LDC teachers used LDC resources, such as the planning process or 
CoreTools, with only and 35% agreeing with this statement. 
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Figure 3.6. Scale-up and sustainability (TL22). 


3.3 Administrator Survey Results 


This section presents the survey results from school principals and assistant principals. 


LDC Participation (A1) and Professional Learning Community (A2). Of the 17 school 
administrators who responded to this survey, 11 (65%) were principals and six (35%) were 
assistant principals. The majority of administrators (41%) reported attending less than one 
quarter of the LDC PLC meetings, 29% attended about one quarter, 12% attended about a half, 
and 12% attended about three quarters. One administrator (6%) reported attending more than 
three quarters. Administrator survey respondents overall reported somewhat similar levels of 
administrator attendance at PLC meetings as did teacher respondents. 


Training and Support (A3-4). Administrators had the opportunity to attend professional 
development sessions, either online or in person, such as Launch Day and administrator 
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meetings. The reported range of meeting attendance was between 0 and 20, with an average of 
attendance of 6 meetings (mean = 5.6). 


Perceptions of LDC were overwhelmingly positive. All administrations (100%) agreed or 
strongly agreed that LDC offered sufficient professional development for teacher leaders, LDC 
offered sufficient professional development opportunities for administrators, and LDC staff 
were able to connect them to additional resources. All administrators (94%, except for one who 
answered N/A to this statement) reported that they were able to reach LDC staff with 
questions. Also, all administrators except for one (94%) agreed that their school had adequate 
technology to access LDC. 


Classroom Observation (A5-6). All administrators reported observing teachers 
implementing LDC at least once during the school year. Half (50%) of the administrators 
observed three or more times, 25% observed two times, and 25% observed one time. Most 
administrators who observed LDC instruction believed that LDC modules were moderately or 
very effective in developing students’ literacy skills (88%). This rating was more positive than 
that from the teacher survey. 


Impact on Teacher Practice (A7). A majority of administrators observed significant 
improvement in all areas of teacher practice probed on, with results similar to teachers’ ratings. 
All administrators but one (94%) believed teachers had improved moderately or a great deal in 
selecting focus standards, identifying necessary skills to complete the writing assignment, 
creating standards-driven writing tasks, creating daily lessons to teach skills to complete the 
writing task, identifying patterns of student understandings or misconceptions, and collecting 
information on student progress in a systematic way. Most administrators (88%) also felt that 
LDC PLC members improved in using evidence of student performance to shape instructional 
decisions. 


Impact on Student Learning (A8). Administrators also were positive, like teachers, about 
the effects of LDC on students. Large majorities of administrators agreed or strongly agreed 
that LDC helped students develop reading skills (93%); improved students’ content knowledge 
(93%); helped students develop speaking and listening skills (93%); increased students’ 
performance on assessments throughout the school year (93%); improved students’ ability to 
complete writing assignments (87%); helped students understand components of the writing 
task (87%); improved students’ writing quality (87%); supported students’ college and career 
ready skills (87%); and helped students’ overall literacy performance (87%). 


Administrator Leadership Role (A9). Almost all administrators felt they played an active 
role in LDC implementation: 93% agreed or strongly agreed that they were able to shape LDC 
implementation at their schools; 93% met regularly with the LDC teacher leader; 93% were 
involved in discussions about differentiating LDC implementation to meet teacher needs; 93% 
made changes to school schedules to accommodate LDC PLC time; 93% allocated resources to 
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ensure LDC team could meet; and 87% led discussions about how to expand LDC 
implementation in future years. 


Alignment (A10). Administrators reflected on how well LDC aligned with other school 
initiatives, programs, and curricula (see Figure 3.7). Here too, administrator responses were 
more positive than those of teachers and teacher leaders. All administrators (100%) agreed or 
strongly agreed that: their school connected LDC to specific schoolwide goals; LDC helped 
teachers create writing assignments to use within current curricula; it complemented other 
initiatives at the school; it was a strategy for implementing state college- and career-ready 
standards; it helped students prepare for state assessments; and they were using LDC to 
implement standards-driven assignments with existing curriculum. Almost all administrators 
(93%) said that LDC helped improve teacher evaluation ratings. 
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Figure 3.7. LDC alignment (A10). 


Survey responses suggest that at least in some sites other initiatives being implemented 
simultaneously created a challenge, with 40% of administrators believing that the time spent on 
LDC interfered with other initiatives, and 60% reporting that it was difficult for teachers to 
focus on LDC because of other competing priorities. 
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Scale-Up and Sustainability (A11). The outlook for LDC implementation in future years 
was generally positive according to administrators, as reported in Figure 3.8. Administrators 
(87%) expected most teachers to continue LDC following year and agreed or strongly agreed 
that teachers and administrators were committed to sustaining LDC. A majority of 
administrators felt that there were new collaborations across grades and/or subjects as a result 
of LDC (67%) and expected the LDC PLC to increase in size in the next year (60%). However, few 
administrators (27%) felt that teachers who were not in the PLC were using the LDC planning 
process and tools. 
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Figure 3.8. Scale-up and sustainability (A11). 


District Support (A12). Administrator ratings of district support appear less positive than 
their ratings in other areas and suggest an area of need. A little over a third (36%) of 
administrators agreed or strongly agreed that district leaders supported the implementation of 
LDC; 29% reported that district leaders visited the school to discuss LDC implementation; 21% 
thought district leaders had a firm understanding of LDC; 21% thought district professional 
development efforts were aligned with LDC; and 14% agreed that district leaders were 
interested in expanding LDC to other schools. 
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3.4 Open-Ended Responses for All Participants 


Each survey respondent was asked for feedback in response to three questions: (1) What 
supports did you find the most useful and why?; (2) What supports were not helpful and why?; 
and (3) In what ways could LDC implementation be improved in your school in the future? 


Below are the summary findings across all three groups of respondents, including 
comments from 179 teachers, 42 teacher leaders (28 of whom were classroom teachers), and 
12 administrators. In reviewing the results in this section, it is important to note that the 
majority of responses to all survey questions was positive. For example, when asked about 
supports that were not helpful, 62 (35%) of teachers, 16 (38%) of teacher leaders, and eight 
(67%) of administrators responded that everything was helpful. 


Professional Learning Community and Program Implementation 


LDC PLC meetings received 50 positive comments. Respondents especially appreciated 
the opportunity to collaborate with other teachers (14 comments) during the LDC PLC. 


The most common suggestion for improving LDC implementation in the future was to 
provide time — either more time or protected time for the PLC. Time was mentioned at least 99 
times, with related requests for meeting more frequently, more time and support to both plan 
and implement modules (not necessarily within the PLC time), more time for collaboration, and 
using time more efficiently. Teachers were generally positive about the content of the PLCs 
(e.g., use of assessment; focus on writing), but some were disappointed with the logistical 
issues (e.g., technology issues, difficulty scheduling meetings, pacing meetings and assignments 
appropriately) and alignment (e.g., standards, existing curriculum, content-area demands). 


At least 27 respondents suggested an earlier start to LDC implementation or more teacher 
buy-in to get the implementation off the ground at their school. A handful of participants 
specifically requested summer training. A few participants requested half or full days of 
training, as opposed to an hour or two during the week, as well. There were 13 comments from 
the respondents who felt that increased teacher buy-in was necessary for LDC to succeed at 
their sites. Ideas for increasing teacher buy-in included teachers having fewer responsibilities so 
that they could focus on LDC implementation and more overall support. Related to this issue of 
teacher buy-in, a few teachers observed that PLC meetings would be more productive if 
teachers were prepared for meetings. Additionally, a few respondents suggested more site 
visits to schools implementing LDC. 


Teacher leaders were viewed favorably with the respondents, with at least eight specific 
mentions of how well the school’s teacher leaders supported teachers. For example, one 
teacher commented, “The most useful supports is meeting with my LDC leader at my school. 
Considering she is creating her own modules and readily available at my school to demonstrate 
what | need done, it has been the most helpful.” However, there were a couple of negative 
mentions from teachers stating that their teacher leader needed more knowledge about LDC. 
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Regarding administrators, the 12 teachers and teacher leaders who mentioned them in the 
open-ended questions wanted more support from them. One teacher wrote that she wished 
for “a little more hands on approach by the administrators and for them to check in more of it.” 
Another teacher asked to “ensure that the administrator understands the demands of having 
LDC as an initiative and what teachers need to be successful.” 


There were 32 suggestions about the breadth of teacher participation in LDC. Nineteen 
suggested including more teachers, and 10 suggested that the entire grade or content area 
department of teachers be included. Additional suggestions for participation included have a 
cross-curriculum or vertical grade level team or having only new teachers participate in LDC. 


Coach (virtual) 


LDC coaches were frequently mentioned, with 140 comments that virtual coaches were 
one of the most useful supports available and 20 comments that the virtual coaches were not 
helpful. These included comments about Zoom videoconference meetings with the entire PLC, 
individual Zoom meetings with coaches, email contact, and written feedback via CoreTools. As 
one teacher wrote, “virtual coaching has been most useful because it allowed our team to 
visually collaborate modules because our coach used split screens and we were given on the 
spot feedback and next steps.” 


Among the positive mentions, respondents praised coaches’ feedback and guidance in 
navigating LDC website content like CoreTools as a whole, the library of modules, and the 
online courses. Respondents appreciated that they could ask questions and receive immediate 
feedback. For example, one teacher wrote, “I found that constant interactions with our coach 
was most effective and useful. She was willing to clarify any misconceptions that | had about 
creating performance based tasks.” They appreciated one-on-one videoconferences with 
coaches and email, and especially appreciated written feedback via CoreTools (23 comments). 
One teacher wrote, “I found the core tools online platform, the virtual coaching especially 
useful because | was able to get instant feedback on what | implemented in the classroom as 
well as get suggestions on how to improve my teacher practice.” 


Respondents who felt virtual coaches were not helpful cited technical problems as the 
most common issue. Across all groups, there were 26 mentions of technology being a problem. 
When the videoconferencing software and hardware did not function properly, participants felt 
there was time wasted. Other negative comments included that the virtual coaches did not 
allow for personalized or differentiated support during the Zoom meetings. Notably, among 
suggestions for improvement, 21 comments were made about requesting more in-person 
coach visits, while only two respondents requested more video conferencing (with one 
respondent wanted more one-on-one video conferences), and one teacher leader requesting 
the same virtual coach for the following year. 


Coach (in-person) 
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Here we report on comments about several types of in-person coaching, including virtual 
coaches’ site visits, assistance from the district liaison and LDC liaison, and professional 
development trainings or workshops. There were 54 comments indicating that in-person 
coaching was one of the most useful supports, and six comments about in-person coaching not 
being helpful. The complaints against in-person coaching was mostly due to scheduling 
conflicts, as teachers mentioned that they were not available to meet with the coach when she 
came to campus. For those teachers who thought in-person coaching was the most useful 
support, they liked the immediate feedback. As mentioned earlier, there were 21 requests for 
more in-person coaching, with some suggesting having a coach observe classroom 
implementation of LDC modules, model implementation, and provide feedback and immediate 
debriefing after an observation. 


LDC professional development trainings or workshops received 18 positive comments, 
seven requests for more workshops. There were 14 negative comments about the LDC 
professional development, in which few respondents thought there was too much information 
presented and that LDC should present clear goals for the program and implementation (e.g., 
scope and research). Ten teachers felt that LDC training and professional development should 
be more differentiated and flexible to allow for teachers with different levels of knowledge to 
benefit from LDC. 


Whether in person or virtually, coaching was the most mentioned support in the open- 
ended responses. Respondents appreciated receiving feedback in any format, whether via 
individual Zoom conferences, during PLC conferences, via email, or the CoreTools platforms. 


CoreTools 


CoreTools received 103 positive comments, and 59 negative comments. Included in this 
count were comments about the content of the LDC library, most of which were positive. 
Participants who liked CoreTools mentioned the wealth of information on the site, including the 
rubrics and accessing mini-tasks and modules. Online courses, part of the CoreTools platform, 
received 12 positive mentions. 


For those participants who struggled with CoreTools, the most commonly cited issues 
were navigation (22 comments), library modules not being user-friendly (8 comments), and 
limited content in the library modules (11 comments). Related to this idea of limited content 
were eight comments that the LDC modules in the library did not meet students’ needs, either 
in terms of reading level, content (e.g., elementary grades, science), or participation in special 
programs (English learners and students with disabilities. The three most commonly cited 
barriers to mastering use of CoreTools were ease of use (i.e., navigation), lack of time to 
explore, and difficulty modifying existing modules. Related to the idea of ease of use, the most 
popular suggestion (9 comments) for improvements called for additional training in navigating 
CoreTools. 
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3.5 Summary of Results 


These surveys captured the responses of 60% of all participants in the district’s second 
year of implementation. In addition to answering closed-ended questions, most of the 
respondents also provided narrative comments about aspects of the program they felt were 
useful as well as those that could use improvement. Generally, respondents provided positive 
feedback. Overall, the survey results suggest the following. 


Across all participant groups, survey responses showed positive attitudes toward LDC. All 
measures of satisfaction or improvement were rated positively by a majority of participants. 


Three quarters of teachers, 84% of teacher leaders, and 87% of administrators agreed 
that LDC helped improve students’ literacy performance. The three most highly rated areas of 
impact on students, according to all three groups, were students’ ability to complete writing 
assignments, content knowledge, and quality of students’ writing. 


LDC coaches received almost unanimous positive feedback, with 98% of teachers and 
100% of teacher leaders reporting that their coaches gave them appropriate and timely 
feedback and support. 


Teacher leaders were almost universally reported to be highly approachable, effective, 
and knowledgeable. 


Almost all teachers reported that their administrators encouraged LDC participation at the 
school, although only 62% of teachers reported receiving feedback from administrators about 
LDC and 40% reported never being observed while teaching an LDC task. Administrators’ 
recollections about their classroom observations were more positive regarding their frequency. 
However, the response rate on the administrator survey was lower, and therefore responses 
are less generalizable to the total population. 


Almost three fourths of teachers felt they had sufficient time to meet during professional 
learning community (PLC) sessions and 81% felt that their administrators allocated resources to 
ensure that they could meet. However, the most frequently requested modification for future 
years was for more time during and outside of PLCs to plan modules, implement, and receive 
feedback about implementation. 


Almost one third of teachers and half of teacher leaders reported meeting in their PLCs 
once every two weeks, the frequency expected by LDC. However, 58% of teachers and 44% of 
teacher leaders reported meeting once a week or more. For teachers, the most cited barrier to 
meeting every week was that PLC members had other priorities that competed with LDC 
participation, and for teacher leaders, the most cited barrier was that PLC time was not 
protected. 


Two thirds of teachers expressed interest in learning more about how to lead LDC 
implementation at their schools in the following year. The majority of teacher leaders (84%) 
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and almost half of administrators (47%) expected their teachers to continue with LDC the 
following year. 


While 83% of teachers agreed that their school had adequate technology to support LDC 
implementation, open-ended responses indicated that successfully conducting Zoom meetings 
was a common complaint. 


The majority of teachers (between 76% to 92%) reported success in nine key areas of LDC 
module development. Teachers were most confident in selecting focus standards, creating the 
writing assignment, identifying skills needed in the module, and making writing assignments 
relevant and engaging, and somewhat less confident in skills involving lesson planning and 
differentiation. 


With regard to their classroom implementation of LDC modules, the majority of teachers 
reported success with all six key areas queried (between 81% to 88%). Teachers were most 
confident with engaging students in complex text, locating evidence of standards in final 
student work, and engaging students in understanding the assignment and rubric. 


Most teachers found CoreTools to be helpful, but about a third of teachers reported that 
the website was not easy to use. The majority of teachers thought that online course materials 
were clear and relevant and provide an opportunity to extend learning when needed. 


The majority of teachers reported improving in skills related to their own practice and 
learning in all seven skills listed (between 73% and 85%). Teachers were most likely to report 
impact in the following skills: selecting focus standards, creating standards-driven writing 
assignments, identifying skills students need in writing assignments, and identifying patterns of 
student understandings or misconceptions. 


Over 80% of teachers agreed that participating in LDC raised their expectations for 
student’s writing, helped them incorporate writing assignments into their existing curriculum, 
and made them more likely to collaborate with other teachers on designing instruction. 


This year’s survey responses were consistently positive across all dimensions and 
participant groups. Suggestions and recommendations provided in this report were primarily 
about adapting to and ameliorating district constraints, such as protected time for meetings 
and technological limitations. 
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4.0 Analyses of LDC CoreTools Data 


In the following section, we report on how participants interacted with LDC’s CoreTools 
online system. We begin by presenting participation rates for key CoreTools activities, including 
creating a user account, viewing modules, editing modules, and commenting on modules. 
Subsequently, we dig deeper into CoreTools viewing, editing, and commenting by sharing 
descriptive statistics for all i3 CoreTools users followed by results broken down by key 
subgroups (including participant role, cohort, school level, and content area taught). Then, we 
compare the level of engagement for i3 CoreTools users who completed and taught a full- 
length module to those users who did not complete and teach a module during the school year. 
Finally, we summarize results for the chapter. 


4.1 CoreTools Activity Participation Rates 


The four key CoreTools activities we examined are creating a user account, viewing 
modules, editing modules, and commenting on modules. Among the 373 CoreTools users, 315 
were classroom teachers (with 34 of those individuals acting as teacher leaders), 19 were 
coaches or coordinators playing the teacher leader role, and 39 of them were administrators. 
As seen in Table 4.1, nearly all participants used CoreTools to at least some degree. Eighty-eight 
percent of all participants created a user account, 76% of participants viewed modules, and 
62% edited modules. Commenting on modules was a less common activity with only 44% of all 


participants doing so. 


Table 4.1 


CoreTools Feature Participation Rates: 2017-2018 


Participant type 


Number of 
participants 


Number 
and % of 
participants 
with user 
accounts 


Number and 
% of 
participants 
who viewed 
a module 


Number and 
% of 
participants 
who edited a 
module 


Number and % 
of participants 
who 
commented on 
a module 


Teachers 


Teacher Leaders 
(classroom teachers) 


Total teachers 


Teacher Leader 
(coach/coordinator)?* 


Administrator (principals 
and assistant principals) 


Total participants 


307 
38 


345 
20 


60 


425 


281 (92%) 
34 (89%) 


315 (91%) 
19 (95%) 


39 (65%) 


373 (88%) 


266 (87%) 
34 (89%) 


300 (87%) 
14 (70%) 


10 (17%) 


324 (76%) 


221 (72%) 
32 (84%) 


253 (73%) 
11 (55%) 


0 (0%) 


264 (62%) 


155 (50%) 
24 (63%) 


179 (52%) 
8 (40%) 


2 (3%) 


189 (44%) 
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In addition to displaying participation rates on key CoreTools activities for all participants, 
Table 4.1 also displays subgroup results for participants playing different roles in the LDC 
implementation. These subgroups mirror the four groups to whom we administered surveys: 
teachers, teacher leaders who teach in the classroom, teacher leaders who are coaches and 
coordinators, and administrators (principals and assistant principals). Teachers and teacher 
leaders had considerably higher participation rates than the principals and assistant principals, 
as expected. Classroom teachers playing the teacher leader role had substantially higher levels 
of engagement than coaches or coordinators playing the teacher leader role. 


Nearly all participants created a CoreTools user account. Significantly, almost two thirds of 
administrators, who generally were not a part of the regular PLC meetings, created user 
accounts. This suggests that most administrators overseeing LDC PLCs had at least interacted 
with the online platform. With that said, less than one fifth of administrators viewed modules in 
the platform, while a large majority of PLC teachers and teacher leaders viewed modules in 
CoreTools. 


We consider editing modules and mini-tasks to be the key indicator of deep engagement 
with the CoreTools module building platform. Our survey results indicated that nearly three 
quarters of participating teachers edited at least one module, compared to 55% of coaches and 
zero principals or assistant principals. 


Overall, adding comments to modules was a much less common activity, with a little over 
half of classroom teachers having commented. Commenting was most common among teacher 
leaders, whose role was most conducive to giving feedback to PLC members. Only a couple 
principals or assistant principals utilized the commenting function. 


4.2 Engagement with Key CoreTools Activities 


In this section, we describe participants’ level of engagement with three key CoreTools 
activities: viewing modules, editing modules, and commenting on modules. Descriptive 
statistics are reported for all participants with CoreTools user accounts, as well as a number of 
subgroups. Those subgroups capture the role the individual played in LDC (teacher, teacher 
leader, and administrator), the level of the school at which the participant worked (elementary, 
K-8, middle, 6-12, or high), the cohort of schools the participant belonged to, and in the case of 
teachers, the content area taught (elementary/multiple subjects, secondary ELA, secondary 
history/social studies, and secondary science). As noted earlier, some participants played 
multiple roles in the intervention, so the teacher and teacher leader groups overlap to some 
degree as they do in our survey analysis. There was also some overlap in sample across 
secondary content areas (for example, some teachers taught both ELA and social studies). 


Descriptive statistics are only reported for groups with five or more members. The 
samples for the viewing, editing, and commenting analyses are the 373 LDC participants with 
CoreTools user accounts in 2017-2018 (see bottom row of Table 4.1). 
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Module viewing. In Table 4.2, we present descriptive statistics on how many times i3 
participants viewed modules, both overall and by subgroup. We present the minimum number 
of views, the maximum number of views, the mean number of views, the standard deviation, 
and the sum of total views across all participants. Overall the average participant viewed 
modules or mini-tasks about 33 times, while the range was from zero views to 1,062 views. 
There were 49 participants with user accounts who did not view any modules (about 13 percent 
of the overall sample). The standard deviation of 82.2 suggests that about two thirds of all 
participants viewed modules between 0 and 115 times (with the remaining third viewing a 
greater number of times). 


Table 4.2 


Descriptive Statistics for the Number of Times a Participant Viewed a Module in CoreTools, by Participant 
Subgroup 


Subgroup n Min Max M SD Sum 

All participants 373 0 1,062 32.8 82.2 12,244 
Participant role 

All teachers 315 0 1,062 37.0 88.3 11,658 

All teacher leaders 53 0 216 40.6 43.3 2,149 

All administrators 39 0 50 2.2 8.2 86 
Cohort 

Cohort 1 Returning Teachers 70 0 1,062 55.8 149.7 3,904 

Cohort 1 New Teachers 65 0 699 45.3 108.2 2,946 

Cohort 2 Teachers 238 0 146 22.7 24.9 5,394 
School level 

Elementary school level participants 140 0 130 25.8 27.3 3,610 

K-8 school level participants 45 0 142 31.9 37.1 1,435 

Middle school level participants 140 0 146 18.8 22.1 2,626 

6-12 school level participants 40 0 1,062 111.7 226.1 4,467 

High school level participants 8 0 34 13.3 11.2 106 


Content area taught 


Taught elementary/multiple subjects 120 0 130 26.0 25.0 3,114 
Taught secondary ELA 30 1 1,062 146.1 252.0 4,383 
Taught secondary social 17 0 315 32.4 73.4 550 


studies/history 
Taught secondary science 8 0 28 14.1 8.5 113 
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As would be expected, teacher leaders on average had the greatest number of views 
among the three participant roles, while administrators viewed by far the least number of 
times. There was considerable variation in viewing behavior across participant subgroups within 
these categories. Cohort 1 teachers viewed modules at higher levels than Cohort 2 teachers. 
Participants in 6-12 schools viewed considerably more modules on average than their peers in 
schools with other grade ranges. Secondary ELA teachers had by far the highest level of 
engagement in terms of module viewing, followed by secondary social studies/history teachers, 
elementary teachers, and finally secondary science teachers. 


Module editing. On average, participants with CoreTools user accounts made almost 16 
edits to modules over the course of the school year. There was a wide range of engagement 
from editing zero times to making 309 edits to modules. One-hundred nine participants with 
CoreTools user accounts (29%) did not do any editing of modules. Two thirds of participants 
edited between zero and 46 times. Table 4.3 reports the descriptive results. 


As with page viewing results, teacher leaders edited more than teachers not playing a 
leadership role. No administrators edited any modules. Cohort 1 teachers edited modules at 
considerably higher levels on average than Cohort 2 teachers. As with module viewing, teachers 
at 6-12 schools and secondary ELA teachers edited at higher levels than their peers. 
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Table 4.3 


Descriptive Statistics for the Number of Times a Participant Edited a Module in CoreTools, by Participant 
Subgroup 


Subgroup n Min Max M SD Sum 

All participants 373 0 309 15.8 30.6 5,884 
Participant role 

All teachers 315 0 309 17.9 32.5 5,648 

All teacher leaders 53 0 110 27.1 25.8 1,436 

All administrators 39 0 0 0.0 0.0 0 
Cohort 

Cohort 1 Returning Teachers 70 0 278 24.8 47.1 1,733 

Cohort 1 New Teachers 65 0 309 21.5 45.2 1,398 

Cohort 2 Teachers 238 0 91 11.6 14.9 2,753 
School level 

Elementary school level participants 140 0 70 11.2 14.8 1,570 

K-8 school level participants 45 0 83 13.9 19.1 627 

Middle school level participants 140 0 91 12.0 17.7 1,673 

6-12 school level participants 40 0 309 48.8 73.1 1,952 

High school level participants 8 0 21 7.8 6.8 62 


Content area taught 


Taught elementary/multiple subjects 120 0 70 9.9 12.8 1,190 
Taught secondary ELA 30 1 309 53.3 82.1 1,600 
Taught secondary social 17 0 162 21.4 38.0 364 


studies/history 
Taught secondary science 8 0 24 10.8 8.9 86 


Module commenting. Half of participants with user accounts made at least one comment 
on a module. Participants commented between one and 66 times, and on average about three 
times. Across participant roles, teacher leaders had the highest level of engagement in 
commenting (about four comments on average). As with viewing and editing, Cohort 1 
participants and secondary ELA teachers showed a higher level of engagement than their peers. 


53 


Table 4.4 


Descriptive Statistics for the Number of Times a Participant Commented on a Module in CoreTools, by 
Participant Subgroup 


Subgroup n Min Max M SD Sum 

All participants 373 0 66 2.9 6.0 1,065 
Participant role 

All teachers 315 0 66 3.2 6.4 1,018 

All teacher leaders 53 0 29 4.1 5.9 215 

All administrators 39 0 9 0.3 1.5 11 
Cohort 

Cohort 1 Returning Teachers 70 0 66 5.3 11.5 373 

Cohort 1 New Teachers 65 0 23 2.6 4.2 169 

Cohort 2 Teachers 238 0 20 2.2 3.4 523 
School level 

Elementary school level participants 140 0 66 3.6 7.7 508 

K-8 school level participants 45 0 27 2.5 4.8 144 

Middle school level participants 140 0 23 1.8 3.6 253 

6-12 school level participants 40 0 29 4.1 7.1 163 

High school level participants 8 0 12 3.4 4.1 27 


Content area taught 


Taught elementary/multiple subjects 120 0 66 3.5 8.0 425 
Taught secondary ELA 30 0 27 6.0 7.5 180 
Taught secondary social 17 0 7 1.2 1.9 21 


studies/history 
Taught secondary science 8 0 8 1.8 2.7 14 


4.3 CoreTools Engagement as an Implementation Variable 


To evaluate the validity of CoreTools engagement as an indicator of LDC implementation, 
we examined the relationship between the three CoreTools engagement measures and module 
implementation. As described in Chapter 6, we use the presence of uploaded student work as 
evidence of teachers having taught the module in their classrooms. Figure 4.1 displays the 
mean number of CoreTools views, edits, and comments for the 134 participants who completed 
and taught LDC modules (as indicated by the upload of student work samples) and their 211 
peers for whom we do not have evidence of module implementation. As shown, teachers who 


54 


completed and taught LDC modules exhibited considerably more engagement with CoreTools 
than their peers, across all three metrics. The results suggest that participants who engage 
deeply with the module building platform are more likely to complete and teach LDC modules. 


Mean CoreTools Views, Edits, and Comments Made by 
Teachers Who Did and Did Not Complete and Teach 
Modules 


70,0 
60,0 
50,0 
40,0 
30,0 
20,0 
10,0 ie 5,7 


0,0 a 


Number of CoreTools Views Number of CoreTools Edits Number of CoreTools Comments 


58,1 


30,0 
18,7 


1,2 


Teachers who completed and taught modules (n=134) 


Teachers who did not complete and teach modules (n=211) 


Figure 4.1. Mean number of CoreTools views, edits, and comments made by teachers who did and did 
not complete and teach modules. 


4.4 Summary of Results 


Descriptive analysis of CoreTools user behavior shows broad engagement with LDC’s 
module building platform but the depth of that engagement varied greatly across users. Nearly 
all 2017-2018 LDC participants, including principals, created a CoreTools user account, and the 
vast majority of teachers in PLCs viewed and edited modules. The average teacher, however, 
only made 16 edits to modules. While this might be interpreted as evidence that teachers’ 
engagement in instructional design was somewhat limited, the CoreTools data does not provide 
any measures of instructional design work conducted offline by individuals or groups of 
teachers. Furthermore, this descriptive analysis provides a broad rather than deep look, and 
does not take into account LDC expectations that teachers will do very little design work on 
their first module and extensive instructional design on their later modules. 


Teacher leaders on average engaged at higher levels than their peers not playing a 
leadership role in the LDC intervention. Cohort 1 returning and new teachers tended to engage 
at higher levels than Cohort 2 teachers. Secondary ELA teachers also engaged at substantially 
higher levels than secondary social studies and science teachers and elementary teachers. 
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5.0 Module Artifact Analysis 


This section presents results for the analysis of modules submitted during the 2017-2018 
school year. We first present overall results across the grade bands. This is followed by separate 
results for the elementary and secondary modules. Finally, we present results of an exploratory 
analysis examining results across time for Cohort 1 teachers who submitted modules across the 
last two school years, the qualitative results, and a summary of the findings. Appendix F has the 
Generalizability Theory results. 


Table 5.1 shows the overall means and standard deviations by grade band for the module 
artifact analysis. When examining the ratings, the elementary modules generally received 
higher mean ratings than did the secondary modules. More specifically, mean ratings for the 
elementary modules ranged from 3.25 to 3.95, indicating that on average all dimensions were 
moderately present or realized. In contrast, mean ratings for the secondary modules were in 
the low to mid threes (3.22 to 3.49). The highest mean for each grade band was for 
Dimension 1, which measures the effectiveness of the writing task. 


Furthermore, as one would hope to find, the average summary scores for the first five 
dimensions were similar to the overall impression ratings (Dimension 6) provided by the expert 
teachers. Similar results were found when examining the frequencies, with between 42% and 
69% of elementary modules and between 38% and 50% of secondary modules receiving ratings 
of four or greater out of a possible five on the six dimensions. Greater detail about the 
frequencies can be found in Appendix F. 


Table 5.1 


Means and Standard Deviations for the Modules 


Dimension Elementary Secondary Overall 
Dimensions n=65 n=76 n=141 
1. Effective writing task 3.95 (1.19) 3.49 (1.14) 3.70 (1.18) 
2. Standards alignment 3.81 (1.10) 3.29 (0.98) 3.53 (1.06) 
3. Fidelity to LDC instruction 3.55 (1.26) 3.38 (1.26) 3.46 (1.26) 
4. Quality instructional strategies 3.25 (1.13) 3.32 (1.28) 3.28 (1.21) 
5. Coherence/clarity of module 3.29 (1.06) 3.22 (1.18) 3.26 (0.95) 
6. Overall impression 3.48 (0.94) 3.32 (0.97) 3.39 (0.95) 
Average (Dimensions 1 to 5) 3.57 (0.92) 3.34 (0.80) 3.45 (0.86) 
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5.1 Analysis of Elementary Modules 


The following section presents descriptive results for the elementary modules 
disaggregated by content area and then by cohort. Results from the generalizability theory 
modules as well as expanded descriptive results for the elementary modules can be found in 
Appendix F. 


Descriptive results by content area. Table 5.2 presents descriptive results for the 
elementary modules by content area. A few comparative observations can be made. First, when 
looking across the content areas, modules generally received higher ratings on the effective 
writing task (Dimension 1) than most other dimensions. Second, the dimensions where 
modules received somewhat lower means varied depending upon the content area. The lowest 
mean for ELA involved Dimension 3, the lowest for science involved Dimension 5, and the 
lowest for social studies involved Dimension 4. Third, the average summary scores for the first 
five dimensions were similar to the overall impression ratings (Dimension 6) provided by the 
expert teachers. Finally, the social studies modules had the largest means on each dimension 
and also had the highest proportion of ratings of four or greater out of a possible five when 
compared to the other content area results. 


Table 5.2 


Means and Standard Deviations for the Elementary Modules by Content Area 


Dimension ELA Science Social Studies 
Dimensions n=34 n=9 n=22 
1. Effective writing task 3.76 (1.21) 3.67 (1.41) 4.36 (1.00) 
2. Standards alignment 3.99 (0.66) 2.44 (1.51) 4.09 (1.11) 
3. Fidelity to LDC instruction 3.15 (1.23) 3.67 (1.22) 4.14 (1.13) 
4. Quality instructional strategies 3.24 (0.92) 2.78 (1.20) 3.45 (1.37) 
5. Coherence/clarity of module 3.32 (0.88) 2.33 (0.87) 3.64 (1.18) 
6. Overall impression 3.38 (0.85) 2.89 (0.93) 3.86 (0.94) 
Average (Dimensions 1 to 5) 3.49 (0.79) 2.98 (1.06) 3.94 (0.92) 


Descriptive results by cohort. Table 5.3 presents descriptive results for the elementary 
modules by cohort: Cohort 1 returning teachers, Cohort 1 new teachers, and Cohort 2 teachers. 
Because of the large differences in sample sizes, we focus our discussion on some general 
trends. First, mean ratings were generally highest for the first two dimensions, which focused 
on the effective writing task and standards alignment. Second, teachers tended to struggle the 
most regarding Dimensions 4 and 5, which rate the quality of instructional strategies and the 
coherence and clarity of the module. Third, in comparing the Dimension 6 ratings to the 
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average for the first five dimensions, it is apparent that expert teachers’ overall impressions 
were fairly aligned for each of the cohort subgroups. Finally, the proportion of ratings of four or 
greater for each cohort subgroup were the greatest for Dimension 1 (70% to 100%). 
Furthermore, less than half of the Cohort 1 Returning and Cohort 2 modules received ratings of 
four or greater for Dimensions 4 and 5. 


Table 5.3 


Means and Standard Deviations for the Elementary Modules by Cohort Grouping 


Dimension Cohort 1 Returning Cohort 1 New Cohort 2 

Dimensions n=10 n=1 n=54 

1. Effective writing task 4.10 (0.99) _ 3.91 (1.23) 
2. Standards alignment 4.10 (0.74) -- 3.73 (1.15) 
3. Fidelity to LDC instruction 3.80 (0.92) -- 3.48 (1.31) 
4. Quality instructional strategies 3.00 (1.05) -- 3.26 (1.14) 
5. Coherence/clarity of module 3.10 (0.99) -- 3.30 (1.06) 
6. Overall impression 3.50 (0.85) -- 3.44 (0.94) 
Average (Dimensions 1 to 5) 3.62 (0.68) -- 3.54 (0.95) 


5.2 Analysis of Secondary Modules 


The following section presents descriptive results for the secondary modules 
disaggregated by content area and then by cohort. Results from the generalizability theory 
modules as well as the expanded descriptive results for the secondary modules can be found in 
Appendix F. 


Descriptive results by content area. Table 5.4 presents descriptive results for the 
secondary modules by content area. While the subgroups vary greatly in size, some general 
observations can still be made. For example, there was very little variation in the ratings for 
ELA, with means ranging from 3.09 to 3.20. In contrast, there were moderate amounts of 
variation in mean ratings for social studies (3.61 to 3.91) and large amounts of variation for 
science (2.56 to 4.00). In addition, means tended to be higher for the effective writing task 
(Dimension 1) and lower for coherence and clarity of the module (Dimension 5). Furthermore, 
as one would hope to find, the average summary scores for the first five dimensions were 
similar to the overall impression ratings (Dimension 6) provided by the expert teachers. Finally, 
when looking at the proportion of ratings of four or greater out of a possible five, percentages 
were generally less than 50% for ELA and science and were greater than 50% for each of the 
social studies dimensions. 
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Table 5.4 


Means and Standard Deviations for Secondary Modules by Content Area 


Dimension ELA Science Social Studies 
Dimensions n=44 n=9 n=23 
1. Effective writing task 3.16 (1.03) 4.00 (1.00) 3.91 (1.20) 
2. Standards alignment 3.18 (0.97) 2.56 (1.01) 3.78 (0.74) 
3. Fidelity to LDC instruction 3.18 (1.32) 3.22 (1.30) 3.83 (1.07) 
4. Quality instructional strategies 3.20 (1.11) 3.11 (1.36) 3.61 (1.53) 
5. Coherence/clarity of module 3.09 (1.05) 2.89 (0.93) 3.61 (1.44) 
6. Overall impression 3.09 (0.83) 3.00 (1.00) 3.87 (1.01) 
Average (Dimensions 1 to 5) 3.16 (0.67) 3.16 (0.79) 3.75 (0.91) 


Descriptive results by cohort. Table 5.5 presents descriptive results for the secondary 
modules by cohort: Cohort 1 Returning, Cohort 1 New, and Cohort 2. In this case, mean ratings 
generally in the low to mid threes for each of the dimensions across the three cohort 
subgroups. The exceptions involve standards alignment (Dimension 2) for the Cohort 1 
Returning modules, and quality instructional strategies (Dimension 4) and coherence and clarity 
of the module (Dimension 5) for the new Cohort 1 teachers. Furthermore, as with the content 
area results, the average summary scores for the first five dimensions were similar to the 
overall impression ratings provided by the expert teachers. Finally, only a small to moderate 
proportion of the modules received ratings that were four or greater for the different 
dimensions for the Cohort 1 returning teachers (23% to 55%), Cohort 1 new teachers (24% to 
52%), and Cohort 2 teachers (42% to 55%). 
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Table 5.5 


Means and Standard Deviations for the Secondary Modules by Cohort Grouping 


Dimension Cohort 1 Returning Cohort 1 New Cohort 2 

Dimensions n=22 n=21 n=33 

1. Effective writing task 3.41 (1.26) 3.33 (1.11) 3.65 (1.08) 
2. Standards alignment 2.96 (1.04) 3.43 (1.08) 3.42 (0.83) 
3. Fidelity to LDC instruction 3.46 (1.26) 3.19 (1.47) 3.46 (1.15) 
4. Quality instructional strategies 3.46 (0.96) 2.90 (1.61) 3.48 (1.20) 
5. Coherence/clarity of module 3.46 (0.86) 2.67 (1.39) 3.42 (1.15) 
6. Overall impression 3.41 (0.73) 3.05 (1.24) 3.42 (0.90) 
Average (Dimensions 1 to 5) 3.34 (0.65) 3.10 (1.04) 3.48 (0.70) 


5.3 Exploratory Analysis of Modules 


As previously noted, in order to examine potential growth among the Cohort 1 teachers 
who continued with LDC in 2017-2018 (returning teachers), ratings were compared for teachers 
who submitted complete modules in both the 2016—2017 and 2017—2018 school years. The 
following presents descriptive results for the last complete module submitted by these 
teachers. Additional descriptive results for the exploratory analyses can be found in Appendix F. 


Table 5.6 presents results for the five elementary teachers and 14 secondary teachers 
who submitted complete modules during both school years. While the sample sizes are too 
small to be significant, a few observations can be made. First, mean ratings were generally 
greater for the modules submitted during the 2017—2018 school year than they were for the 
2016—2017 school year. Among the elementary modules, the exceptions involved Dimensions 4 
and 5, which measure quality instructional strategies and coherence and clarity of the module. 
In contrast, the only dimension where the means for the secondary modules went down 
involved standards alignment (Dimension 2). When comparing the average summary scores for 
the first five dimensions with Dimension 6, means were somewhat closer in alignment at the 
first time-point than at the second time-point. 
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Table 5.6 


Means and Standard Deviations for the Exploratory Analysis of Modules 


Dimension 2016-2017 2017-2018 Change 

ELEMENTARY (n = 5) 

1. Effective writing task 4.00 (1.73) 4.20 (0.84) 0.20 
2. Standards alignment 3.40 (1.52) 4.00 (0.71) 0.60 
3. Fidelity to LDC instruction 2.60 (1.14) 4.00 (0.71) 1.40 
4. Quality instructional strategies 3.40 (1.34) 2.80 (1.10) -0.60 
5. Coherence/clarity of module 3.20 (1.92) 3.00 (1.22) -0.20 
6. Overall impression 3.20 (1.48) 3.40 (0.89) 0.20 
Average (Dimensions 1 to 5) 3.32 (0.87) 3.60 (0.71) 0.28 
SECONDARY (n = 14) 

1. Effective writing task 3.29 (1.20) 3.57 (1.22) 0.29 
2. Standards alignment 3.14 (1.17) 2.93 (1.14) -0.21 
3. Fidelity to LDC instruction 3.14 (1.29) 3.29 (1.59) 0.14 
4. Quality instructional strategies 3.00 (1.24) 3.93 (0.62) 0.93 
5. Coherence/clarity of module 2.93 (1.38) 3.79 (0.80) 0.86 
6. Overall impression 3.07 (1.14) 3.64 (0.84) 0.57 
Average (Dimensions 1 to 5) 3.10 (0.99) 3.50 (0.81) 0.40 
OVERALL (n = 19) 

1. Effective writing task 3.47 (1.35) 3.74 (1.15) 0.26 
2. Standards alignment 3.21 (1.23) 3.21 (1.13) 0.00 
3. Fidelity to LDC instruction 3.00 (1.25) 3.47 (1.43) 0.47 
4. Quality instructional strategies 3.11 (1.24) 3.63 (0.90) 0.53 
5. Coherence/clarity of module 3.00 (1.49) 3.58 (0.96) 0.58 
6. Overall impression 3.11 (1.20) 3.58 (0.84) 0.47 
Average (Dimensions 1 to 5) 3.16 (0.94) 3.53 (0.77) 0.37 


5.4 Qualitative Results 


Expert raters also participated in debriefings to shed light on their ratings and suggest 
potential ways to improve ratings in the future. These debriefings were conducted at the end of 
each week of ratings. The following presents key findings concerning Dimensions 1 through 5, 
as well as our expert raters’ overall impressions. 
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Dimension 1: effective writing task. While a majority or the modules received higher 
ratings of four or five on this dimension, the writing task was not always realized. Modules that 
received lower ratings for this category tended to have a writing task that was either overly 
generic or too verbose for the expert rater to understand how it would be taught. In addition, 
the student background section was not used in a consistent manner, with some teachers using 
it to provide background about their students and others using this component as an outline of 
what they said to students. Finally, the extension option was rarely used and tended to provide 
little detail. 


Dimension 2: standards alignment. Elementary teachers were more consistent than were 
secondary teachers in their specification of standards for the modules. When providing lower 
ratings for this dimension, the expert raters noted a few issues. First, modules tended to have 
reading standards, but were less likely to have focused writing standards to support the essays 
being written by the students. Second, some of the adapted modules included multiple-grade 
standards that did not align with the grade level listed. Because of the limited backgrounds and 
reflections provided it was difficult for the expert raters to determine whether this was done 
purposefully or not. Third, some of the adapted modules included standards from states other 
than California. 


Dimension 3: fidelity to LDC module instruction. Expert raters noted some overarching 
trends regarding modules that received higher versus lower levels of fidelity. First, those 
modules with stronger fidelity tended to have stronger standards alignment. Second, modules 
that showed lower fidelity tended to be missing sufficient transition to writing standards and 
mini-tasks. Third, many modules included a mini-task on writing body paragraphs, but failed to 
provide instruction on how to write introductory and closing paragraphs. 


Dimension 4: quality instructional strategies. Raters noted two trends that impacted 
their ratings of the instructional strategies. First, mini-tasks were not always well organized, 
making it difficult to determine what was being taught on a daily basis. Second, multiple issues 
with student work were encountered. This included attachments that were hand written and 
difficult to read as well as work that appeared to be from a previous administration of the 
module. Expert raters also noted that it was easier to measure quality when teachers included 
the completed rubrics with the student work samples. 


Dimension 5: coherence and clarity of module. The expert raters found the student work 
to be vital to measuring the coherence and clarity of the modules. This was especially true for 
modules that included multiple work samples at varying levels as well as those that included 
pre-writing and not just final papers. 


Dimension 6: overall impressions. Feedback on the overall quality of the modules tended 
to mimic the ratings of coherence and clarity. Modules that received higher ratings on 
Dimension 5 also received higher ratings on Dimension 6. In part, this had to do with the rater’s 
ability to discern how well the module was actually realized as evidenced in the student work 
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samples, student background, and teacher reflection sections. In addition, modules that 
seemed ready to be implemented as written tended to be considered higher in overall quality. 


5.5 Summary of Results 


Based on the primary analyses, teachers during the 2017—2018 school year were at least 
moderately successful in their development of high quality modules. When examining the 
overall samples, mean ratings were in the low to high threes for the elementary modules and in 
the low to mid threes for the secondary modules. In addition, when looking at the individual 
ratings, the largest proportion of modules received ratings of three or four, indicating that the 
modules were moderately to sufficiently present or realized. The only exception involved 
Dimension 1, involving the effective writing task, for the elementary sample with 43% of 
modules receiving a rating of five, or fully present or realized. 


Modules submitted by elementary and secondary teachers also tended to have different 
strengths. More specifically, elementary teachers performed strongest in how they set up the 
modules (Dimensions 1 and 2), with means in the high threes and more than 60% of the 
modules receiving ratings of four or five. In contrast, while secondary teachers still performed 
stronger regarding Dimension 1, they performed slightly better in their fidelity to LDC 
instruction (Dimension 3) than they performed in regards to their standards alignment 
(Dimension 2). 


Differences were also found when examining the subgroup results for the elementary and 
secondary teachers. For example, elementary teachers tended to perform stronger in how they 
set up the modules than in how they carried them out. More specifically, means were greatest 
for Dimensions 1 and 2 for the ELA modules as well as each of the cohort subgroups (Cohort 1 
returning, Cohort 1 new, and Cohort 2). Similarly, means were greatest for Dimensions 1 and 3 
for the science and social studies modules. In contrast, many of the secondary subgroups 
performed stronger on some of the less foundational dimensions, such as the quality of 
instructional strategies. 


An exploratory analysis was also conducted to examine growth in performance for the 
five elementary and 14 secondary Cohort 1 returning teachers who submitted complete 
modules with student work during both the 2016-2017 and 2017-2018 school years. While 
significance cannot be determined because of the small sample sizes, mean ratings did improve 
for the majority of dimensions. The only exceptions involved the quality of instructional 
strategies (Dimension 4) and coherence and clarity (Dimension 5) for the elementary teachers, 
and standards alignment (Dimension 2) for the secondary teachers. Finally, the greatest 
increases in mean ratings were for fidelity to LDC instruction (Dimension 3) among the 
elementary teachers and quality instructional strategies (Dimension 4) for the secondary 
teachers. 
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6.0 Fidelity of Implementation Analysis 


This chapter describes the descriptive results of the fidelity of implementation to the LDC 
model at NYCDOE schools. UCLA CRESST and LDC collaborated to create a fidelity matrix to 
measure schools’ fidelity of implementation across four key components of the LDC model. 
Within the matrix, each key component is measured via multiple indicators, with 14 indicators 
total across these four key components. 


In this chapter, we will first present the overall results on school- and program-level 
fidelity scores for all the indicators. We will then provide more detailed descriptive results with 
sections for each of the four key components and sub-sections for each of the 14 fidelity 
metrics. 


6.1 School- and Program-Level Fidelity of Implementation Scores 


Table 6.1 provides a summary for how LDC schools performed across the four key 
components and their associated 14 indicators of fidelity of implementation. As noted in 
Chapter 2, schools were rated on four-point scales (from 0 to 3) on each indicator. At the school 
level, adequate implementation was defined as a score of at least 2 (with the exception of 
principal mini-task observation where school-level fidelity was set at a score level of 1). For the 
indicators under Key Component 2, program-level fidelity was met if half or more of the schools 
met fidelity. For all of the other indicators, program-level fidelity was met if three quarters of 
schools met fidelity. 


As displayed in Figure 6.1, there was great variation across schools in their success in 
meeting implementation thresholds. Schools met fidelity on a range between zero and nine 
indicators, with an average of just over four indicators. No schools met fidelity on more than 
nine indicators, suggesting that even high performance schools have room for growth. 


Overall, the analysis revealed that a majority of the schools were not meeting the 
adequate implementation thresholds. Adequate implementation at the program level was met 
on just two of the 14 indicators: coach comments on modules and teacher perception of the 
helpfulness of coach written feedback on modules. Nearly half of the schools also met the 
threshold for teacher attendance at weekly PLC meetings and teacher leader attendance at 
quarterly in-person teacher leader meetings, although those proportions were not sufficient for 
the program to be labeled as having adequate implementation on those indicators. Less than 
40% of schools met the thresholds for the other ten indicators. For fidelity to be met on a key 
component, the fidelity matrix requires that fidelity be met on each of the indicators for that 
key component. As a result, the analysis for 2017-2018 concludes that fidelity was not met for 
the four key components. 
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Table 6.1 
Summary Table of School- and Program-Level Fidelity Scores by Indicator in 2017-2018 


% of 
schools %of % of % of % of 
with schools schools schools schools Program 
Key no at at at at Level met 
component Indicator data LevelO Level1 Level 2 3 fidelity? 

Key Teacher attendance at weekly PLC 8 34 9 21 28 No 
Component 1: meetings 
pommen : Amount of time spent on LDC during 2 19 64 13 2 No 
Banning. tine common planning time 
for LDC 
Professional Exposure to LDC LEARN content 0 100 0 0 0 No 
Learning during first instructional cycle 
Community 
with Exposure to LDC LEARN content 0 98 0 0 2 No 
Synchronous during second instructional cycle 
Coach Support Perceived effectiveness of 15 25 4 25 32 No 

engagement in PLC on teacher 

competencies 
Key Coach comments on modules 6 26 11 13 43 Yes 
COMPONENT: Coach formative peer review on 6 85 2 4 4 No 
Asynchronous 

modules 
Support from 
LDC Coaches Teacher perception of the 15 17 11 9 47 Yes 

helpfulness of coach written 

feedback on modules 
Key Module editing 0 75 9 6 9 No 
Omponent: Module implementation 0 91 6 0 4 No 
Teacher 
Implementatio 
n Activities 
Key Frequency of coach/teacher leader 2 17 47 19 15 No 
Component 4: monthly meetings 
pedocelle Administrator attendance at 0 32 36 17 15 No 
Support at : Pee 
Different quarterly in-person administrator 

meetings 
Levels 

Teacher leader attendance at 0 15 36 34 15 No 

quarterly in-person teacher leader 

meetings 

Principal mini-task observation 15 43 6 9 26 No 
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Frequency Distribution of Schools Reaching Adequate 
Implementation on Different Numbers of Fidelity Indicators 
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Figure 6.1. Frequency Distribution of Schools Reaching Adequate Implementation on Different Numbers 
of Fidelity Indicators 


6.2 Key Component 1: Common Planning Time for LDC Professional Learning 
Community with Synchronous Coach Support 


Schools varied greatly in their level of fidelity on the five indicators of Key Component 1, 
as shown in Figure 6.2. Schools were most successful with regards to Perceived Effectiveness of 
Engagement in PLC on Teacher Competencies and Teacher Attendance at Weekly PLC Meetings 
with a little over half (57%) and a little under half (49%) meeting fidelity on those metrics 
respectively. Only 15%met fidelity on Amount of Time Spent on LDC During Common Planning 
Time. Meeting fidelity on the indicators of exposure to LDC LEARN content was nearly uniformly 
challenging for schools, with no schools reaching the adequate implementation threshold for 
the first instructional cycle and just one school reaching the threshold for the second 
instructional cycle. 
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Schools Reaching Different Fidelty Levels for Each Key 
Component 1 Indicator 
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Figure 6.2. Number of Schools Reaching Different Fidelity Levels for Each Key Component 1 Indicator 


Teacher Attendance at Weekly PLC Meetings. PLC Reflection data show that schools 
varied greatly in terms of the number of times their PLCs met over the course of the 2017-2018 
school year. The number of sessions ranged from zero to 25 times, with PLCs on average 
meeting 11.8 times. This is important context for the interpretation of the teacher attendance 
at weekly PLC meetings fidelity indicator. The indicator measures the proportion of teacher 
who regularly attended sessions, but does not incorporate the amount of times that the PLC 
met across the year. Although success in the two facets of attendance seems to be somewhat 
correlated, there were cases where teacher attendance rates were high but the school met a 
below average number of times. And likewise there were cases where schools met a large 
number of times but individual attendance rates were poor. 


Attendance rates across the full population of PLC participants varied a great deal, ranging 
from 6 to 100%and averaging 79%. Not surprisingly, teacher leaders who also taught in 
classrooms (n=33) attended at higher average rates than classroom teachers not playing a 
leadership role (n = 243): 82%compared to 79%. Teacher leaders in out of classroom 
assignments (n=21) attended at lower rates (69%). The adequate implementation threshold for 
individual teacher attendance was 80%or greater, and 230 out of 339 PLC participants (68%) 
reached this threshold. While a majority of teachers in a majority of schools reached this 
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threshold, there was a very substantial minority of schools that experienced low attendance 
rates. 


Amount of Time Spent on LDC During Common Planning Time. This metric is based on 
PLC reflection data for the subset of PLC sessions that coaches joined either digitally or in 
person. Overall, teacher leaders reported that 146 sessions were under 45 minutes long (23%), 
372 sessions were between 45 and 59 minutes (60%), 84 sessions were between 60 and 74 
minutes (14%), and 22 sessions were between 75 and 90 minutes (4%). 


School-level scores were based on the modal (most common) response on the PLC 
reflection form. For 10 of the schools, that response was under 45 minutes, for 34 of the 
schools it was 45-59 minutes, for seven of the schools it was 60-74 minutes, and for one of the 
schools it was 75-90 minutes. Because of the nature of the data, we cannot ascertain the exact 
length of the sessions. As such it is possible that many of the sessions recorded as 45-59 
minutes came quite close to the desired 60 minutes of PLC time. Nevertheless, according to the 
thresholds set, a majority of the schools did not meet fidelity on this indicator. It may make 
sense for LDC to update its instrument to collect more fine-grained data on this indicator. 


Exposure to LDC LEARN Content During First and Second Instructional Cycles. Here we 
report on the two indicators measuring PLC participants’ engagement with the LEARN online 
course content (instructional courses that help LDC participants learn how to navigate an LDC 
instructional cycle). Of the 366 teachers and teacher leaders participating in LDC PLCs in 2017- 
2018, 249 participants (68%) did not view any LEARN content through their user accounts. The 
remaining 117 participants viewed at least one session in one instructional cycle. And a subset 
of that group (n=64) viewed at least one session in a second instructional cycle. Overall the full 
population of teachers and teacher leaders on average were exposed to 9% of the LEARN 
sessions in the first instructional cycle and 5% of the LEARN sessions in the second instructional 
cycle. 


It is possible that teachers were not complying with instructions to engage with the 
LEARN content within their own user accounts but were still exposed to some of the content in 
the group setting of the PLC. Nevertheless, even if we examine only those participants with any 
exposure to LDC in the first instructional cycle (n=117), that smaller group still only was exposed 
to an average of 29%of the sessions. Most teachers were not being exposed to the majority of 
LEARN sessions in their first instructional cycle, with even lower results in the second 
instructional cycle. 


Due to the overall low fidelity at the teacher level, nearly all of the schools scored at the 
lowest fidelity level for both exposure to LEARN content indicators. The only exception was one 
school that performed at a high level on the second instructional cycle. The overall low 
performance of schools on this indicator, however, masks variation across the schools in 
performance. Most significantly, 21 out of 54 of the schools (39%) had zero teachers with any 
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exposure to LEARN content via their user accounts. On the high end, a small number of schools’ 
PLC members were exposed on average to 40-45% of the LEARN content. 


Perceived Effectiveness of Engagement in PLC on Teacher Competencies. This indicator is 
based on the 210 teachers who provided data on question 30 in the teacher survey (see 
Appendix A), which asks teachers to report how much their skills had improved in a number of 
areas aligned with LDC core competencies. Here we base fidelity levels on an index averaging 
the seven question 30 items, which are each measured on a 1 to 4 scale. The threshold for 
adequate implementation at the teacher level is an index score of 3, corresponding to the 
survey response reporting moderate improvement in the skill area. Across the whole sample, 
17 teachers had scores of 1 to less than 2 (no to a little improvement), 40 had scores of 2 to less 
than 3 (a little to moderate improvement), 104 had scores of 3 to less than 4 (moderate to 
great improvement), and 49 had scores of 4 (great deal of improvement). A little less than three 
quarters of respondents in the sample reached the adequate implementation threshold score 
of 3 on the index. 


There was a great deal of variation in survey responses both within and across schools. In 
12 schools, half or fewer of the teachers reported a moderate effect on the competencies. On 
the other hand, 100%of respondents in 17 schools reported a moderate or greater effect. The 
variation can be clearly seen in Figure 6.2. 


6.3 Key Component 2: Asynchronous Support from LDC Coaches 


As with Key Component 1, there was a great deal of variation across schools in their level 
of fidelity on the three Key Component 2 indicators, as shown in Figure 6.3. Nevertheless, a 
majority of schools met the adequate implementation thresholds for coach comments on 
modules and teacher perception of the helpfulness of coach written feedback on modules, and 
as aresult the program met fidelity on these two metrics. Coach peer review was not a broadly 
used tool, with only a handful of schools meeting fidelity on this indicator. Together, the 
findings suggest that the program was fairly successful with regard to coaches providing useful 
feedback to LDC participants, although expectations were not being met for the use of the 
more comprehensive peer review tool as a formative feedback mechanism. 
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Number of Schools Reaching Different Fidelity Levels for Each 
Key Component 2 Indicator 
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Figure 6.3. Number of Schools Reaching Different Fidelity Levels for Each Key Component 2 Indicator 


Coach Comments on Modules. Results on this indicator are based on the universe of 
modules linked to a LEARN instructional cycle. In total there were 210 modules linked to 
courses across 50 of the 53 schools, with the number of linked modules per school ranging from 
one to seven. Those modules in turn received between 0 and 21 comments from coaches (26 
modules received 0 comments). An adequate level of feedback on a linked module was defined 
as having received two or more coach comments, and 156 out of the total 210 modules (74%) 
met that threshold. The proportion of modules within schools that met the two comment 
threshold varied a great deal across schools, which accounts for the variation in school level 
fidelity scores as seen in Figure 6.3. 


Coach Formative Peer Review on Modules. Results on this indicator are based on the 
universe of teachers and teacher leaders who co-authored at least one linked course. That 
population includes 246 participants across 50 schools. At the teacher level, fidelity was defined 
as having received at least one peer review from a coach. Only 46 teachers received at least 
one peer review from their coach, and therefore only 8%of the schools met the school-level 
fidelity threshold. 


Teacher Perception of Helpfulness of Coach Written Feedback on Modules. Across 212 
teacher survey respondents, 168 teachers (79%) reported using written feedback on modules 
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and finding it moderately to very helpful. Across schools, the proportion of teachers who 
reported at least moderate usefulness ranged from 13 to 100%(with data not available for eight 
schools as a result of a O%survey response rate at those locations). A majority of schools met 
the adequate implementation threshold, and therefore the program as a whole met fidelity on 
this metric. 


6.4 Key Component 3: Teacher Implementation Activities 


Here we report on indicators related to teachers’ engagement with the process of 
designing and implementing LDC modules. As shown in Figure 6.4, schools were more 
successful in terms of meeting thresholds for the proportion of their teachers who engaged 
deeply in the design process by editing key portions of modules, with about 15%of schools 
meeting fidelity on this indicator. In only two schools did three quarters or more of teachers 
implement two modules, as measured by the uploading of student work samples. 


Number of Schools Reaching Different Fidelity Levels for Each 
Key Component 3 Indicator 
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Figure 6.4. Number of Schools Reaching Different Fidelity Levels for Each Key Component 3 Indicator 


Module Editing. As outlined in the fidelity matrix, teacher-level fidelity levels are built in a 
step ladder fashion. To meet fidelity level 1, a teacher needed to edit at least one task ina 
module. To meet fidelity level 2 (adequate implementation), a teacher needed to reach the 
level 1 threshold and edit either standards or text in at least one module. To reach fidelity level 
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3, a teacher had to meet the previous requirements and also edit either skills and mini-tasks or 
the rubric. 


Out of 366 teachers and teacher leaders, 66%failed to meet the threshold of editing the 
teaching task in one module and therefore implemented at fidelity level of zero. Seven 
participants (2%) edited a task, but did not edit standards or texts, and therefore were at 
fidelity level 1. A total of 117 participants (32%) met the adequate implementation threshold, 
with 39 (11%) scoring at level 2 (edited standards or texts but not skills/mini-tasks or rubric) 
and 78 (21%) scoring at level 3 (edited standards or text AND skills/mini-tasks or rubric). As a 
result of just one third of participants overall meeting the teacher-level adequate 
implementation threshold, only 15%of schools met the school level threshold for adequate 
implementation. 


Module Implementation. For this indicator, our sample is the 346 classroom teachers 
who participated in LDC in 2017-2018 (as the metric involves classroom implementation, out of 
classroom staff are excluded). The upload of student work serves as a proxy for whether the 
teacher implemented a module in her classroom. The number of teachers with uploaded 
student work ranged from zero to eight with a mean of 0.66. Two hundred eleven teachers 
(61%) didn’t upload student work to any modules. Seventy-four (21%) uploaded student work 
to one module, and 45 (13%) uploaded to two modules. Sixteen teachers (5%) uploaded 
student work to 3 or more modules. Overall, less than 20%of teachers met the adequate 
implementation threshold; with these teachers being spread across many schools, only two 
schools met fidelity on the indicator. 


6.5 Key Component 4: Leadership Support at Different Levels 


As depicted in Figure 6.5, there was a great deal of variation across schools on each of the 
Key Component 4 metrics. Depending on the metric, between a third and a half of schools met 
fidelity. These results were not positive enough for fidelity to be met at the program level on 
any of the metrics. 
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Number of Schools Reaching Different Fidelity Levels for Each 
Key Component 3 Indicator 
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Figure 6.5. Number of Schools Reaching Different Fidelity Levels for Each Key Component 3 Indicator 


Frequency of Coach/Teacher Leader Monthly Meetings. Teacher leaders reported 
meeting with their coach between one and 22 times with a mean of 7.6 times, according to PLC 
reflection data. The adequate implementation threshold was set at nine or more coach/teacher 
leader meetings across the school year, and only 18 of 53 schools (34%) met that threshold. 
Twenty-five of 53 schools (48%) met between four and eight times (low implementation) and 
nine (17%) met less than four times (very low implementation). 


Administrator Attendance at In-Person Administrator Meetings. Administrators had the 
opportunity to attend three events. Administrators in eight schools (15%) attended all three 
events, in nine schools (17%) attended two events, in 19 schools (36%) attended one event, and 
in 17 schools (32%) attended zero events. With the adequate implementation threshold set at 
two events, 17 schools (32%) met the threshold. 


Teacher Leader Attendance at Quarterly In-Person Teacher Leader Meetings. Teacher 
leaders also had the opportunity to attend three events. Teacher leaders in 8 schools (15%) 
attended all three events, in 18 schools (34%) attended two events, in 19 schools (36%) 
attended one event, and in eight schools (15%) attended zero events. With the adequate 
implementation threshold set at two of three events, 26 out of 53 (49%) schools met the 
threshold. 
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Principal Mini-Task Observation. Thirty-six percent of teacher survey respondents 
reported never having been observed by an administrator when teaching an LDC mini-task, 
28%reported being observed once, 10%reported being observed twice, and 16%reported being 
observed three or more times. Despite almost two thirds of respondents reporting having been 
observed, less than half of schools met the adequate implementation threshold of three 
quarters of teachers having been observed. Fidelity was therefore not met for the program as a 
whole on this metric. 


6.6 Summary of Results 


In summary, the fidelity matrix analysis revealed that none of the schools met fidelity 
requirements on the four key components in 2017-2018 in New York. The ability of PLCs to set 
aside common planning time that worked for all teachers varied across schools. The frequency 
of meetings ranged broadly, as did the attendance rates of the participants. About half of 
schools were able to ensure that at least three quarters of their teachers attended PLC 
meetings at high rates. When PLCs did meet, they most often met for between 45 and 59 
minutes, although many sessions also lasted the desired 60 minutes or more. LDC intended for 
teachers to be exposed to key online course content within PLC time, but by and large this did 
not happen. Many participants did not view any LEARN content in their user accounts (although 
they may have been exposed to the content via a peer or coach’s account), and those that did 
tended to view a minority of sessions in the instructional cycle. The evidence suggests that 
many participants did not embark on a second instructional cycle. Given the wide variety of 
experiences that schools and teachers had with common planning time, it is not surprising that 
teachers reported a variety of attitudes with regards to the effectiveness of PLC in building 
teacher skills. 


Results on asynchronous support were a bright spot in the fidelity analysis. A majority of 
schools met expectations for coaches providing written feedback on modules, and teachers 
reported that this feedback was useful in a majority of schools (LDC met fidelity on these two 
metrics). Coach peer review on the other hand was infrequently utilized. 


Results on module editing suggest that many teachers were not heavily engaged in the 
design process, with two thirds of teachers not having edited any teaching tasks. Those that did 
edit the teaching task tended to also edit either standards or texts, and were also more likely 
than not to reach the stage of editing skills/mini-tasks and the rubric. Only about 40%of 
teachers uploaded student work to at least one module, and just 18%uploaded student work to 
two or more modules. 


Results on leadership support varied greatly across schools. There were clearly schools 
where leadership support was high, with teacher leaders and administrators attending 
professional development opportunities, teacher leaders regularly meeting with coaches, and 
administrators observing LDC instruction. But a majority of schools did not meet expectations in 
these areas. 
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Variation across schools was substantial, with some schools meeting fidelity on as many 
as seven to nine indicators, and on the other end, some schools struggling with nearly every 
aspect of implementation. No patterns emerged for cohort and school level subgroups with 
Cohort 1 and Cohort 2 schools and elementary and secondary schools meeting similar levels of 
fidelity, despite great variation within each of the subgroups. Analysis of implementation in 
2018-2019 will yield additional information on how fidelity changes over time. 
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7.0 Student Outcome Analysis 


This section presents the student outcome analysis we conducted to evaluate LDC’s 
impact on student learning in the 2017-2018 school year. As described earlier, our LDC teacher 
sample included both elementary and middle school teachers from two cohorts of LDC schools 
in NYCDOE. Some of the LDC teachers in Cohort 1 schools were in their second year of LDC 
implementation (having participated in 2016-2017) and some of the teachers newly joined the 
existing PLCs in 2017-2018. LDC teachers from Cohort 2 school were all in their first year of 
participating in LDC in 2017-2018. Separate sampling and analyses were conducted for three 
groups of teachers: Cohort 1 returning middle school teachers, Cohort 2 elementary school 
teachers, and Cohort 2 middle teachers. 


We could not conduct analyses for the other groups of teachers because of sample sizes. 
More specifically, we could not conduct meaningful analyses of the impact of Cohort 1 
returning or Cohort 1 new elementary school teachers due to the small number of students 
who were taught by these Cohort 1 teachers and for whom we have their 2015-2016 baseline 
data. There was also too small a number of students taught by Cohort 1 new middle school 
teachers in 2017-2018 to conduct meaningful analyses. 


We begin this chapter by describing the process we used to define the LDC student 
samples for each analysis and to construct matched comparison samples. We then present 
descriptive statistics for the treatment and comparison groups. Finally, we report the estimated 
impact of LDC on students as measured by their New York State Assessment ELA scores. 


7.1 LDC Sample and the Matching Process 


As described earlier, our LDC teacher sample included both elementary and middle school 
teachers in the study district. Separate sampling and analyses were conducted for these two 
groups of teachers, as described in Section 2.5 of this report (Analytical Approaches). 


Cohort 1 returning middle school sample. The eligible LDC sample for the Cohort 1 
returning middle school analysis includes all students (1) who were enrolled in one school 
campus for the entire 2017-2018 school year under the instruction of at least one of the 
participating LDC teachers in their second year of LDC participation, and (2) for whom prior 
achievement scores, outcome year achievement scores, and demographic data were available. 
Achievement and demographic data were used in the matching process. 


The 2017-2018 Cohort 1 returning middle school sample prior to the CEM process 
included ten schools, 26 Cohort 1 returning middle school teachers who consented to 
participate in the evaluation study, and their 1,626 students. After the student-level matching, 
our final Cohort 1 returning middle school LDC sample included 1,511 students and the same 
number of teachers and schools prior to matching (see Table 7.1). 
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Prior to matching, the potential middle school comparison sample consisted of 514 
schools, 9,871 teachers, and 151,030 students. This comparison sample was substantially 
reduced during the first stage of matching, which identified up to five schools that closely 
resembled each of the ten LDC middle schools. After student-level CEM, a workable analytic 
comparison sample consisted of 43 schools, 753 teachers and 1,511 students. 


Table 7.1 
Before and After Matching Sample Sizes: Cohort 1 Returning Middle School Analysis 


LDC sample Comparison sample 
Stage Schools Teachers Students Schools Teachers Students 
Stage 1 
Before matching 10 26 1,626 514 9,871 151,030 
After matching 10 26 1,626 50 911 12,602 
Stage 2 
After matching 10 26 1,511 43 753 1,511 


Cohort 2 elementary sample. As with the Cohort 1 analysis, the eligible LDC sample for 
the Cohort 2 elementary school analysis includes all students (1) who were enrolled in one 
school campus for the entire 2017-2018 school year under the instruction of at least one of the 
participating LDC teachers, and (2) for whom prior achievement scores, outcome year 
achievement scores, and demographic data were available. Achievement and demographic 
data were used in the two stage matching process. 


Table 7.2 
Before and After Matching Sample Sizes: Cohort 2 Elementary School Analysis 


LDC sample Comparison sample 
Stage School Teacher Student School Teacher Student 
Stage 1 
Before matching 14 57 1,351 781 10,722 120,696 
After matching 14 57 1,351 70 1,962 9,722 
Stage 2 
After matching 14 54 1,258 65 538 1,258 


77 


As reported in Table 7.2, the resulting LDC sample included 14 schools, 57 teachers, and 
1,351 students prior to the CEM process. After the CEM student-level matching, our final 
Cohort 2 elementary LDC sample was reduced to 1,258 students. Prior to matching, the 
potential comparison sample consisted of 781 schools, 10,722 teachers and 120,696 students. 
This comparison sample was substantially reduced during the first stage of matching, which 
identified up to five schools that most closely resembled each of the 14 LDC schools. After 
student-level CEM, a workable analytic comparison sample of 65 schools, 538 teachers, and 
1,258 students was constructed. 


Cohort 2 middle school sample. The eligible LDC sample includes all students (1) who 
were enrolled in one school campus for the entire 2017-2018 school year under the instruction 
of at least one of the participating LDC teachers, and (2) for whom prior achievement scores, 
outcome year achievement scores, and demographic data were available. Achievement and 
demographic data were used in the matching process. Again, we employed a two stage 
matching process to select schools and students. 


As shown in Table 7.3, the resulting Cohort 2 LDC middle school sample included 10 
schools, 31 teachers, and 1,396 students prior to the CEM process. After the CEM student-level 
matching, our final secondary LDC sample was reduced to 1,285 students. 


Table 7.3 
Before and After Matching Sample Sizes: Cohort 2 Middle School Analysis 


LDC sample Comparison sample 
Schools Teachers Students Schools Teachers Students 
Stage 1 
Before matching 10 31 1,396 516 9,897 156,773 
After matching 10 31 1,396 50 968 14,651 
Stage 2 
After matching 10 31 1,285 41 651 1,285 


Prior to matching, the potential comparison sample consisted of 516 schools, 9,867 
teachers and 156,773 students. This comparison sample was substantially reduced during the 
first stage of matching, which identified up to five schools that closely resembled each of the 10 
LDC schools. After student-level CEM, a workable analytic comparison sample of 41 schools, 
651 teachers, and 1,285 students was constructed (see Table 7.3). 
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7.2 Descriptive Results on the Matched Analytic Samples 


Tables 7.4, 7.5 and 7.6 present the characteristics of the LDC student and comparison 
students in the final analytical samples for the Cohort 1 returning middle school, Cohort 2 
elementary, and Cohort 2 middle school analyses respectively. Treatment and comparison 
samples matched very closely. Exact matching was achieved on some variables, and all 
demographic variables were within four percentage points. We used spring 2018 New York 
State Assessment ELA scores as our outcome measures in all analyses. The Cohort 1 analysis 
used the spring 2016 New York State ELA Assessment as one of the matching variables while 
the Cohort 2 analyses used the spring 2017 ELA assessments. 


For the prior achievement matching variable, we standardized New York State 
Assessment scores at each grade level relative to district performance, based on the district 
mean and standard deviation for the ELA test at each grade level. Standardizing scores in this 
way enables us to easily compare our samples’ performance relative to the district’s and to 
compare scores across grades and years more easily. A standardized scale score of zero, for 
example, indicates that the student scored at the mean for all other students in the district who 
took the same test. A standardized scale score of 1.0 meant that the student scored one 
standard deviation higher than the district mean. Conversely, a standardized scale score of -1.0 
indicated that the student scored one standard deviation lower than the district mean. 


The final LDC Cohort 1 middle school student sample after matching was comprised 
largely of students eligible for free or reduced price lunch (see Table 7.4). Slightly more than 
half of students were Hispanic and about a third were black, and the sample was well 
distributed across the three grade levels in the baseline year (sixth, seventh, and eighth grade in 
2017-2018). Special education students represented about one-quarter of this sample, while 
English language learners represented slightly more than 10%. Mean performance on the prior 
year assessments was more than one-third of a standard deviation lower than the district-wide 
performance levels in both mathematics and ELA. 


The final Cohort 2 LDC elementary student sample (see Table 7.5) was also more than one 
half Hispanic and about a third black. Most students were eligible for free and reduced priced 
lunches. The sample was quite evenly distributed between students in third (51.8%) and fourth 
grade (48.2%) in the prior year. English language learners represented slightly more than 15% 
of this sample, while special education students represented slightly less than one quarter of 
the sample. In addition, mean performance on the prior year ELA assessments was about one- 
quarter of a standard deviation lower than the district-wide performance level, while mean 
performance on the prior year math assessment was about one-third of a standard deviation 
lower than the district-wide performance level. 


As shown in Table 7.6 the final Cohort 2 LDC middle school student sample included a 
large combined proportion of Black (51.2%) and Hispanic students (34.0%), and a large majority 
of students with low socioeconomic backgrounds (80.8%). Slightly over one half of the sample 
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included students that were in fifth grade (26.4%), and sixth grade (25.1%) in the prior year, 
with the remainder (48.5%) in seventh grade in the prior year. English language learners 
represented more than one tenth of this sample, while less than one quarter of the sample 
included special education students (23.8%). In addition, mean performance on the prior year 
assessment for LDC students was substantially lower when compared to district-wide 
performance levels in mathematics and ELA. 


Table 7.4 


2015-2016 Student Characteristics of the Cohort 1 Returning Middle School Treatment and Comparison 
Groups After Matching 


Treatment group Comparison group 
Student characteristic (n = 1,511) (n = 1,511) 

Race/ethnicity 

Hispanic (%) 53.5 53.5 

Black (%) 32.2 32.2 

Asian (%) 3.5 3.5 

White (%) 9.9 9.6 

Other (%) 0.9 1.2 
Female (%) 53.6 53.6 
Special programs status 

Poverty (%) 87.2 87.2 

English language learner (%) 10.8 10.8 

Special education (%) 25.6 23.6 
Student prior achievement 

Mean baseline year mathematics Z score -0.406 -0.391 

Mean baseline year ELA Z score -0.334 -0.326 
Class & teacher characteristics 

Mean baseline ELA Z score of current -0.370 -0.348 

peers 

Teacher years of experience 8.0 7.2 
Grade level at baseline year 

Grade 4 in 2015-2016 (%) 34.1 34.1 

Grade 5 in 2015-2016 (%) 37.0 37.0 

Grade 6 in 2015-2016 (%) 28.9 28.9 
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Table 7.5 


2016-2017 Student Characteristics of the Cohort 2 Elementary School Treatment and Comparison Groups 
After Matching 


Treatment group Comparison group 
Student characteristic (n = 1,258) (n = 1,258) 

Race/ethnicity 

Hispanic (%) 56.0 56.0 

Black (%) 35.6 35.6 

Asian (%) 3.0 3.5 

White (%) 3.8 3.7 

Other (%) 1.5 1.2 
Female (%) 50.9 52.6 
Special programs status 

Poverty (%) 85.0 86.3 

English language learner (%) 15.3 12.2 

Special education (%) 22.9 20.4 
Student prior achievement 

Mean baseline year mathematics Z score -0.320 -0.331 

Mean baseline year ELA Z score -0.247 -0.237 
Class and teacher characteristics 

Mean baseline ELA Z score of current -0.244 -0.245 

peers 

Teacher years of experience 12.7 12.2 
Grade level at baseline year 

Grade 3 in 2016-2017 (%) 51.8 51.8 

Grade 4 in 2016-2017 (%) 48.2 48.2 
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Table 7.6 


2016-2017 Student Characteristics of the Cohort 2 Middle School Treatment and Comparison Groups 
After Matching 


Treatment group Comparison group 
Student characteristic (n = 1,285) (n = 1,285) 

Race/ethnicity 

Hispanic (%) 34.0 34.0 

Black (%) 51.2 51.2 

Asian (%) 11.5 7.5 

White (%) 2.4 5.1 

Other (%) 0.9 2.2 
Female (%) 48.2 48.2 
Special programs status 

Poverty (%) 80.8 81.6 

English language learner (%) 13.6 11.6 

Special education (%) 23.8 27.8 
Student prior achievement 

Mean baseline year mathematics Z score -0.540 -0.542 

Mean baseline year ELA Z score -0.409 -0.419 
Class & teacher characteristics 

Mean baseline ELA Z score of current peers -0.434 -0.388 

Teacher years of experience 10.9 10.0 
Grade level at baseline year 

Grade 5 in 2016-2017 (%) 26.4 26.4 

Grade 6 in 2016-2017 (%) 25.1 25.1 

Grade 7 in 2016-2017 (%) 48.5 48.5 


7.3 Outcome Analysis Results: Cohort 1 Middle School Sample 


Analyses were conducted at the middle school level, where the norm was student 
exposure to multiple teachers. Students could be exposed to anywhere between zero and 
fourteen different LDC teachers in the 2017—2018 school year. Using a multiple membership 
multiple classification (MMMC) design, each observation at Level 1 represented one student, 
which was linked to the ELA, social studies/history, and science teachers that the student was 
exposed to during the year. Weights across teachers for each student summed to a unity (1). 
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Two different approaches were used to model the LDC treatment intervention variable as 
a fixed effect at the student level. The first model was dosage dependent, and took into 
account variation in middle school students’ level of exposure to LDC teachers. In this approach, 
the treatment variable was structured as a continuous response variable between zero and 
one, based on exposure to LDC teachers for ELA, social studies/history and/or science. In 
contrast, the second approach was modeled as dosage independent, and considered any 
student exposed to any LDC intervention teacher in at least one subject to be a treated 
individual. In this latter approach the treatment variable was dichotomous, coded as one for 
LDC treated students and zero for comparison students. 


In Table 7.7 we present results of both the dosage dependent and dosage independent 
models on middle school students’ ELA performance in 2017-2018. As can be seen, model 
results for the LDC effect on student outcomes are in the positive direction, but are not 
Statistically significant for either model. In other words, neither analysis provided sufficient 
evidence to conclude that students taught by LDC teachers performed better on the ELA test 
than did their matched peers in the comparison group. 


The significant effects of the covariates on student performance also were similar under 
the two models and were in the expected directions. Prior ELA performance was the strongest 
predictor and prior mathematics performance also helped explain the outcome. In addition to 
prior achievement, each of the race/ethnicity variables and three demographic variables helped 
predict performance: students in special education performed at lower levels than students not 
in special education, females performed at significantly higher levels than males, and students 
with low socio-economic status performed at lower levels than did their peers with higher 
socio-economic status. 
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Table 7.7 


Cohort 1 Returning LDC Middle School Teacher Effect Estimates on 2017-2018 NYS ELA Performance, 
Dosage-Dependent and Dosage-Independent Models 


Variables 


Dosage-dependent model 
coefficient (SD) 


Dosage-independent model 
coefficient (SD) 


Level 2 - LDC teacher treatment 


Level 1 - student characteristics 


Hispanic 

Black 

Asian 

Other 

Poverty 

Female 

English language learner 
Special education 

Grade 7 

Grade 8 

Content Exposure Weeks 
Teacher Experience 


Baseline Peer ELA Z score 


Baseline year mathematics Z score 


Baseline year ELA Z score 


0.025 (0.131) 


-0.123 (0.049)* 
-0.202 (0.054)* 
0.159 (0.074)* 
-0.260 (0.119) * 
-0.129 (0.036)* 
0.218 (0.025)* 
-0.027 (0.043) 

-0.118 (0.031)* 
0.081 (0.041) 

0.137 (0.045)* 
0.011 (0.006) 

-0.002 (0.004) 

0.221 (0.034)* 
0.181 (0.019)* 
0.558 (0.021)* 


0.001 (0.052) 


-0.123 (0.049)* 
-0.202 (0.054)* 
0.159 (0.074)* 
-0.260 (0.119) * 
-0.129 (0.036)* 
0.218 (0.025)* 
-0.027 (0.043) 

-0.118 (0.031)* 
0.081 (0.041) 

0.138 (0.045)* 
0.011 (0.006) 

-0.002 (0.004) 

0.220 (0.034)* 
0.181 (0.019)* 
0.558 (0.021)* 


Note. Based on the dosage-dependent model, the average treated student received a 0.313 treatment dosage. 
Because of this, using the dosage dependent model we estimate an average treatment on the treated (ATT) at 
(0.313 * 0.025) = 0.008. 

*o = .05. 


7.4 Outcome Analysis Results: Cohort 2 Elementary Sample 


As with middle school students, elementary students were typically associated with more 
than one core content teacher; the average treatment student was linked to 2.6 core content 
teachers and the average comparison student was also linked to 2.6 teachers. For this reason, 
and to be consistent with the larger middle school analysis, we employed a multi-membership 
multiple classification (MMMC) design for the elementary analysis. Two different approaches 
were used to model the LDC treatment intervention variable as a fixed effect at the student 
level. The first model was dosage dependent, which took into account some variation in 
elementary students’ level of exposure to LDC teachers. In this approach the treatment variable 
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was structured as a continuous response variable between zero and one, based on exposure to 
LDC teachers for ELA, social studies/history, and/or science. In contrast, the second approach 
was modeled as dosage independent and considers any student exposed to any LDC 
intervention teacher via at least one subject as a treated individual. In this latter approach, the 
treatment variable was dichotomous, coded as one for LDC treated students and zero for 
comparison students. The outcome variable for both models was students’ 2017 New York 
State assessment scores in ELA. For technical reasons of evaluating effect sizes, New York State 
outcome assessment scale scores were standardized to the study sample. 


As can be seen in Table 7.8, model results for the LDC effect on student outcomes are in 
the negative direction, but are not statistically significant for either model. In other words, 
neither analysis provided sufficient evidence to conclude that students taught by LDC teachers 
performed at levels different than their matched peers in the comparison group. The effects of 
the covariates on student performance were similar in direction to those in the Cohort 1 middle 
school model, but in this analysis the race/ethnicity and poverty coefficients were not 
significant, and the English Language Learner coefficient was significant. The effect for grade 
level was not large enough to meet our criteria for inclusion in the model presented below. 
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Table 7.8 


Cohort 2 LDC Elementary School Teacher Effect Estimates on 2017-2018 NYS ELA Performance, Dosage- 
Independent Model 


Variables Dosage Dependent 
model coefficient (SD) 


Dosage independent 
model coefficient (SD) 


Level 1 - LDC teacher treatment 
Level 2 - Student characteristics 

Hispanic 

Black 

Asian 

Other 

Poverty 

Female 

English Language Learner 


Special education 


-0.048 (0.066) 


-0.077 (0.073) 
-0.110 (0.074) 
0.015 (0.094) 
-0.028 (0.121) 
-0.056 (0.039) 
0.079 (0.025) * 
-0.160 (0.040) * 
-0.154 (0.036) * 


-0.003 (0.034) 


-0.076 (0.073) 
-0.110 (0.074) 
0.015 (0.094) 
-0.029 (0.121) 
-0.056 (0.039) 
0.079 (0.025) * 
-0.161 (0.040) * 
-0.154 (0.036) * 


0.001 (0.004) 
0.100 (0.039) * 
0.235 (0.022) * 0.234 (0.022) * 
0.593 (0.022) * 0.593 (0.022) * 


0.001 (0.004) 
0.099 (0.039) * 


% teachers average years of experience 
Baseline peer ELA Z score 
Baseline year mathematics Z score 


Baseline year ELA Z score 


Note. Based on the dosage-dependent model, the average treated student received a 0.642 treatment dosage. 
Because of this, using the dosage-dependent model we could estimate an average treatment on the treated (ATT) 
at (0.642 * -0.048) = -0.031. 

*p =.05. 


7.4 Outcome Analysis Results: Cohort 2 Middle School Sample 


Similar analyses to those conducted for Cohort 1 middle schools were conducted for the 
Cohort 2 middle school sample. At the middle school level, the norm was student exposure to 
multiple teachers. As discussed in detail in Chapter 2, students could be exposed to anywhere 
between zero and twelve different LDC teachers. Using a MMMC design, each observation at 
Level 1 represented one student, which was linked to the ELA, social studies/history and 
science teachers that student was exposed to during the year. Weights across teachers for each 
student summed to a unity (1). 


In Table 7.9 we present results of models that are both dosage dependent and dosage 
independent. Dosage dependent model results indicate an LDC effect in a positive direction 
that approached but did not obtain statistical significance on the student outcome when the p- 
value is set at 0.05. The dosage independent model was also in the positive direction and did 
not yield a statistically significant LDC effect. Once again, covariates were in the expected 
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direction, with in this case, gender, English Language Learner status, special education status 
being significant predictors of ELA achievement. 


Table 7.9 


Cohort 2 LDC Middle School Teacher Effect Estimates on 2017-2018 NYS ELA Performance, Dosage- 
Dependent and Dosage-Independent Models 


Variables Dosage-dependent model Dosage-independent model 
coefficient (SD) coefficient (SD) 
Level 2 - LDC teacher treatment 0.202 (0.108) 0.078 (0.051) 


Level 1 - student characteristics 


Hispanic 0.107 (0.067) 0.107 (0.067) 
Black 0.021 (0.066) 0.021 (0.066) 
Asian 0.113 (0.074) 0.112 (0.074) 
Other -0.086 (0.116) -0.088 (0.116) 
Poverty -0.056 (0.032) -0.055 (0.032) 
Female 0.130 (0.024)* 0.130 (0.024)* 
English language learner -0.195 (0.044)* -0.196 (0.044)* 
Special education -0.167 (0.032)* -0.167 (0.032)* 
Grade 7 0.039 (0.043) 0.037 (0.043) 
Grade 8 0.080 (0.039)* 0.081 (0.040)* 
Teacher Experience 0.003 (0.004) 0.003 (0.004) 
Baseline Peer ELA Z score 0.092 (0.037)* 0.096 (0.037)* 
Baseline year mathematics Z score 0.206 (0.020)* 0.206 (0.019)* 
Baseline year ELA Z score 0.653 (0.021)* 0.653 (0.021)* 


Note. For the dosage dependent model, since the average treatment student received a 0.400 treatment dosage 
we could estimate an average treatment on the treated (ATET) at 0.40 * 0.202 = 0.081. 
Ky - 

p=.05. 


7.5 Summary and Interpretation of Results 


Here we summarize the quasi-experimental results and provide a lens by which the reader 
can contextualize the magnitude of the results. In Figures 7.1 and 7.2, we present the dosage- 
dependent effects for each of the three analyses. Figure 7.1 depicts the estimated impacts of 
LDC in the three samples on students exposed to LDC teachers in all three major content areas: 
ELA, social studies/history, and science. The effect sizes for these estimates can be best 
understood as the estimated impact of LDC under ideal conditions. Figure 7.2 depicts the 
estimated impact of LDC in the three samples on the average observed student, who had 
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considerably less exposure to LDC teachers in her core content classes, particularly in the 
middle school context. 


0.25- 
Teacher Cohorts 
— S Cohort 1 Middle School Teachers (n=26) 
: S Cohort 2 Elementary Teachers (n=54) 
ES cohort 2 Middle School Teachers (n=31) 
-0.25- 


Figure 7.1. Treatment Effect on 2017-2018 New York State Assessment ELA Scores with 95% 
Confidence Interval for Students with Full LDC Dosage, by Cohort 
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ES Cohort 2 Middle School Teachers (n=31) 
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Figure 7.2. Treatment Effect on 2017-2018 New York State Assessment ELA Scores with 95% 
Confidence Interval for Students with Average LDC Dosage, by Cohort 
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As can be seen in the figures, the confidence intervals all cross the zero line, and therefore 
the estimates are not statistically significant at the 95 percent level. The lower bound of the 
confidence intervals around the estimates for the impact of Cohort 2 middle school teachers is, 
however, just below zero, suggesting that the estimates were not far off from being statistically 
significant at the 95 percent level. 


It is important to note that results in this report are exploratory. The evaluation’s main 
confirmatory research questions focus on teachers in their second year of implementation, 
pooling teachers and students from both cohorts. Confirmatory results will depend partially on 
New York State outcome scores in spring of 2019. Future confirmatory analyses will shed more 
light on the impact of LDC in New York City schools. 
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8.0 Summary of Findings 


This annual report examines LDC following two years of implementation in Cohort 1 
schools and one year of implementation in Cohort 2 schools in New York City Department of 
Education. We summarize these results organized by the three categories of evaluation 
questions we listed in Chapter 1 of this report. 


8.1 Program Characteristics and Implementation 


Consistent with prior year results, participants reported overwhelmingly positive attitudes 
toward LDC and its implementation at their schools. Teachers generally appreciated the 
opportunity to collaborate with colleagues, and nearly uniformly praised their LDC coaches. 
Prior to 2017-2018 implementation, LDC made further refinements to the content, sequencing, 
and delivery of CoreTools’ instructional modules with the intent of streamlining the learning 
process for PLC members; this effort seems to have been successful as open-ended responses 
in our current survey did not reference instructional content as a problem. 


In 2017-2018, local PLC leadership was assigned to a Teacher Leader rather than a Project 
Liaison. The role was strengthened by providing a stipend to the individual, and principals and 
assistant principals were no longer permitted to play the role. Again, these program changes 
seem to have been effective. PLC members overwhelmingly reported that their teacher leaders 
were supportive, knowledgeable, and helpful. Teacher leaders themselves also reported high 
satisfaction with support from coaches, professional development offerings, and how the 
Teacher Leader role allowed them to be instructional leaders in their schools. 


Multiple sources of evidence suggest that implementation was largely faithful to the 
intended structure of LDC. Most PLCs met at least every other week as they were expected to 
do. Coaches provided feedback through multiple mechanisms, and PLC members reported 
appreciating this feedback. Analysis of program data suggest that nearly all participants were 
engaging with the module building platform, although that engagement did vary greatly across 
individuals. Module analysis suggests that the materials adapted and created by PLC members 
varied in levels of completion, however, and based on presence of uploaded student work, a 
large proportion of participants did not end up implementing their modules in the classroom. 


Encouraging developments were the increase in average quality of modules based on our 
exploratory analysis of modules by Cohort 1 teachers who completed modules in both years, 
and an increase inthe number of modules teachers completed in CoreTools. The sample size 
for the growth analysis was, however, small, and we anticipate that with lower attrition in 
Cohort 2, we will be able to learn more about change in module quality within teachers over 
time. 
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8.2 Contextual Factors and Implementation 


As noted above, coaches and Teacher Leaders were almost universally praised by teachers 
participating in LDC. Teachers also generally reported that their school administrators were 
supportive of the program, although administrators’ level of participation in PLC meetings and 
observation of LDC instruction varied greatly across the sample, with most schools not meeting 
thresholds on key fidelity indicators. Survey respondents were less confident that district-level 
administrators were knowledgeable about and supportive of the LDC program in their schools. 


As in prior years, adequate time was a frequently cited concern, with some respondents 
hoping for more collaborative time, more prep time, and/or time before the start of the year. 
Technology also remained a problem for some participants. Some respondents were frustrated 
with technical problems that marred the experience of zoom meetings, and some respondents 
simply would have preferred in-person rather than digital meetings with coaches. On a positive 
note, respondents seemed to report less difficulty with the CoreTools platform than in prior 
years, which may relate to both increased skills on the part of Cohort 1 participants, and 
refinements made by LDC to the platform. 


8.3 Program Impacts 


Based on survey results, LDC was perceived to have positive impacts on a range of both 
teacher practices and student skills. Teachers in general felt that LDC had improved their 
instructional planning and pedagogical skillsets and was also helping to promote collaboration 
between teachers. Module analysis suggests that overall mean module quality was higher in 
2017-2018 than 2016-2017. An exploratory look at modules created by LDC teachers who 
participated in both year 1 and year 2 also suggests improvement of quality over time. 


Sufficient sample size was available to conduct quasi-experimental tests of the impact of 
LDC on students taught by Cohort 1 returning middle school teachers, and by Cohort 2 
elementary and middle school students. No statistically significant effects were found for any of 
the three analyses, although the effect for Cohort 2 middle school teachers was positive and 
approaching significance. 
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Appendix A: 
Teacher Survey and Responses 


2017-2018 
LDC Teacher Survey 


LDC Participation 


T1. Please select your school from the drop down box. 
Teachers are skipped to T3 if they teach in an elementary school. 


T2a. Inthe current school year (2017-18), how many classes did you teach? 


| | classes n = 140, Mean = 4.64, Range: 1-20 


T2b. In how many of these classes did you use LDC modules and/or mini-tasks? 


| | classes n=140, Mean = 2.87, Range: 0-13 


T2c. In what content areas did you use LDC modules and/or mini-tasks? 


T2d. In what grades did you use LDC modules and/or mini-tasks? 


T3. Prior to the current school year (2017-18), did you have any experience with LDC? 


(n = 231) 
Yes 52 teachers (22.5%) 
No y Skip to T5a 179 teachers (77.5%) 
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T4. How many of the following did you teach prior to the current school year (2017-18)? 


| | LDC modules n= 52, Mean = 2.40, Range: 0-5 


| | LDC mini-tasks, outside of modules n=52, Mean = 3.60, Range: 0-20 


Professional Learning Community and Teacher Collaboration 


T5a. Did you participate this year in a Professional Learning Community (PLC) at least partly 
focused on implementing LDC in your school? 


(n = 203) 
Yes @] kip to Te 186 teachers (91.6%) 
No 17 teachers (8.4%) 


T5b. Did you use any LDC tools in your instructional planning or classroom instruction this 


year? 
(n=17) 
Yes @ Skip to sc 15 teachers (88.2%) 
No 2 teachers (11.8%) 


T5c. Why did you choose not to use any LDC tools in your instructional planning or 
classroom instruction this year? 


[Survey ends here for respondents answering question T5c] 
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T5d. What LDC tools did you use during the current school year? Select all that apply. 


(n = 15) 
CoreTools online platform to access existing modules or mini-tasks 11 teachers (73.3%) 
CoreTools online platform to design modules or mini-tasks 9 teachers (60.0%) 
LDC online courses 4 teachers (26.7%) 
Modules or mini-tasks given to me by other teachers in my school 6 teachers (40.0%) 
Other (please specify) 2 teachers (13.3%) 


[Survey ends here for respondents answering question T5d] 


T6. About how often did your LDC PLC meet? 


(n = 186) 
Less than once a month 1 teacher (0.5%) 
Once a month 28 teachers (15.1%) 
Every other week 56 teachers (30.1%) 
Once a week y Skip toT8 94 teachers (50.5%) 
Twice a week or more often i> SkiptoT8 7 teachers (3.8%) 


T7. What were the primary barriers preventing your LDC PLC from meeting weekly? Select 


all that apply. 
(n = 85) 

PLC time was not protected. 12 teachers (14.1%) 
PLC members had limited interest in attending meetings. 2 teachers (2.4%) 
School administrator did not make it a priority. 6 teachers (7.1%) 
Teacher Leader did not provide sufficient organizational support. 1 teacher (1.2%) 
Not enough teachers participated. 3 teachers (3.5%) 
PLC members had other priorities that compete with LDC participation. | 52 teachers (61.2%) 
Other (please specify) 26 teachers (30.6%) 
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T8. About how often did you have informal discussions (as opposed to scheduled meetings) 


about LDC with teachers in your LDC PLC? 


(n = 213) 
Less than once a month 30 teachers (14.1%) 
Once a month 39 teachers (18.3%) 
Every other week 38 teachers (17.8%) 
Once a week 64 teachers (30.0%) 
Twice a week or more 42 teachers (19.7%) 


T9. On average, how long did your school’s LDC PLC meetings typically last? 


(n = 213) 
Less than 45 minutes 42 teachers (19.7%) 
45 to 59 minutes 124 teachers (58.2%) 
60 to 74 minutes 41 teachers (19.2%) 
75 minutes or more 6 teachers (2.8%) 


Teacher Training and Support 


T10. How effective was your LDC PLC in the following areas? 


Not A little Moderately Very 
effective effective effective effective 
Creating an environment in which teachers were ad aI} 83 (39.0%) 116 
comfortable working together (n = 213) (0.5%) (6.1%) ess (54.5%) 
Fostering an environment where teachers 1 17 116 
alae their instructional plans with colleagues (0.5%) (8.0%) 79 (37.1%) (54.5%) 
(n = 213) 
4 17 89 103 
Allowi to sh tudent k(n = 213 
owing space to share student work (n ) (1.9%) (8.0%) (41.8%) (48.4%) 
Helping teachers to improve their LDC al 23 91 98 
instructional plans (n = 213) (0.5%) (10.8%) (42.7%) (46.0%) 
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T11. How would you rate each of the following aspects of the online course material (in the 
Learn tab in LDC CoreTools) that your coach used or directed you to use? 


Poor Fair Good Excellent 
a 34 106 66 
Clarity of inf ti ted (n = 213 
arity of information presented (n ) (3.3%) (16.0%) (49.8%) (31.0%) 
: . 3 31 118 61 
Relevance of information presented (n = 213) (1.4%) (14.6%) (55.4%) (28.6%) 
16 48 101 48 
E f = 213 
a (7.5%) (22.5%) (47.4%) (22.5%) 
Usefulness of resource documents (e.g., LDC 
Pitfall Checklist, CCSS Mental Markers, etc.) - S a gi 
(n = 213) (3.3%) (18.3%) (48.4%) (30.0%) 
i 47 110 44 
U i = 
sefulness of videos (n = 213) (5.6%) (22.1%) (51.6%) (20.7%) 
Degree to which course material helped teachers 8 40 108 57 
to create and/or adapt LDC modules (n = 213) (3.8%) (18.8%) (50.7%) (26.8%) 
Opportunity to extend learning when needed or 8 29 111 65 
desired (n = 213) (3.8%) (13.6%) (52.1%) (30.5%) 


T12a. Overall, were you able to get the feedback and support you needed from your LDC 
coach (through written feedback in LDC CoreTools, or coaching and modeling in your 


LDC PLCs) to plan, teach, reflect on, and revise LDC modules)? 


(n = 212) 


Yes 209 teachers (98.6%) 


No 3 teachers (1.4%) 


T12b. Did your LDC coach provide written feedback on your module(s) in LDC CoreTools in a 


timely manner? 


(n = 212) 


Yes 210 teachers (99.1%) 
No 2 teachers (0.9%) 
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T13. Outside of the PLC meetings with your LDC coach, please indicate whether you used 
each of the following types of coach support, and how helpful you found these types 


of support. 
Used 
Did not 
nee Not A little Moderately Very 
helpful helpful helpful helpful 

Written feedback in LDC CoreTools from 
your LDC coach (in the comments 16 1 27 76 92 
areas and/or via the teacher work rubric) (7.5%) (0.5%) (12.7%) (35.8%) (43.4%) 
(n = 212) 
One-on-one Zoom video conference and/or 24 2 Ze 64 100 
call with your LDC coach (n = 212) (11.3%) (0.9%) (10.4%) (30.2%) (47.2%) 
Email or phone communication with your 28 al 21 72 90 
LDC coach (n = 212) (13.2%) (0.5%) (9.9%) (34.0%) (42.5%) 
Other (n = 212) 133 2 9 23 45 
Please specify: (62.7%) (0.9%) (4.2%) (10.8%) (21.2%) 


Module Creation 


T14. During the current school year (2017-18), how many LDC modules did you individually 
or collaboratively adapt from existing modules (e.g., modules you created in a prior year 
and/or modules found in the LDC Library in CoreTools)? 


| | Adapted modules 


n= 184, Mean = 2.08, Range: 0-10 


T15. During the current school year (2017-18), how many LDC modules did you create, either 
individually or with colleague(s)? Only include modules built from scratch, not those 


adapted from existing modules in the LDC library. 


| | New modules 
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n= 184, Mean = 1.39, Range: 0-20 


T16. How did members of your PLC collaborate to create LDC modules? 
Check all that apply. 


(n = 234) 


112 teachers (47.9%) 
145 teachers (62.0%) 
42 teachers (17.9%) 
13 teachers (5.6%) 


Modules were created by individual teachers. 


Modules were created by teams of two or more teachers. 


Modules were created by the PLC as a whole. 


Other (please specify) 


T17. Please indicate to what extent you were able to do each of the following when creating 


LDC modules. 


Not at all Alien a eee 
extent extent 

Select focus standards for a writing 1 15 75 120 
assignment (n = 211) (0.5%) (7.1%) (35.5%) (56.9%) 
Create a standards-driven writing 1 21 75 114 
assignment (n = 211) (0.5%) (10.0%) (35.5%) (54.0%) 
Select high quality, complex texts and other A 7 83 97 
materials to engage students in deeper 
learning (n = 211) (1.9%) (12.8%) (39.3%) (46.0%) 
Identify the skills students need to develop 2 yy) 88 99 
to complete a writing assignment (n = 211) (0.9%) (10.4%) (41.7%) (46.9%) 
Create daily lessons to teach the skills a 10 34 88 79 
student needs to complete a writing 
assignment (n = 211) (4.7%) (16.1%) (41.7%) (37.4%) 
Differentiate instruction by incorporating 
multiple ways of thinking, various levels of 12 38 88 73 
complexity, and multiple modalities (5.7%) (18.0%) (41.7%) (34.6%) 
(i= 271) 
Plan for a variety of methods to assess 6 8 102 75 
student progress (e.g., rubrics and/or mini- 
task scoring guides) (n = 211) (2.8%) (13.3%) (48.3%) (35.5%) 
Assess the quality of writing assignments 
and/or instructional plans using Peer 5 33 92 81 
Review/Curriculum Alignment Rubric (e.g. 
Task Pitfalls Checklist, rubric indicators) (2.4%) (15.6%) (43.6%) (38.4%) 
(n = 211) 
Make a writing assignment relevant and 6 0 91 94 
engaging for students (n = 211) (2.8%) (9.5%) (43.1%) (44.5%) 
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T18. 


TIS, 


T20. 


Classroom Implementation 


How many total LDC modules did you teach during the current school year (2017-18)? 


| | Modules 


n= 211, Mean = 2.48, Range: 0-20 


Outside of modules, approximately how many individual LDC mini-tasks did you teach 
during the current school year (2017-18)? 


| | Mini-tasks n= 211, Mean = 3.65, Range: 0-50 


Please indicate to what extent you were able to do each of the following activities when 


teaching LDC modules. 


Matar sim imAtiteisienl cic alee oe 
extent extent 
Engage students in understanding the 8 19 97 87 
assignment and its rubric (n = 211) (3.8%) (9.0%) (46.0%) (41.2%) 
Engage students in accessing complex text 8 21 98 84 
for the purpose of the assignment (n = 211) (3.8%) (10.0%) (46.4%) (39.8%) 
Systematically collect information about 10 30 93 78 
students’ progress (n = 211) (4.7%) (14.2%) (44.1%) (37.0%) 
Provide feedback to students using 13 22 90 86 
assignment rubrics (n = 211) (6.2%) (10.4%) (42.7%) (40.8%) 
Locate evidence of standards in final 7 19 92 93 
etadesit work on the writing assignment (3.3%) (9.0%) (43.6%) (44.1%) 
(n = 211) 
Use evidence of student progress on 9 7 93 32 
standards to modify subsequent instruction 
(4.3%) (12.8%) (44.1%) (38.9%) 


(n = 211) 
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T21. Toward the beginning of the school year, did you ’find and teach” a module from CoreTools? 


(n = 211) 
Yes 130 teachers (61.6%) 
No y Skip to T24 81 teachers (38.4%) 


T22. What was the name of the Find and Teach module? 


T23. Did you make any adjustments to the Find and Teach module? 


(n = 130) 
Yes 100 teachers (76.9%) 
No 30 teachers (23.1%) 


T24. What module did you adapt, refine, and/or develop most during the current school year 
(2017-18)? This module is typically one you worked on after the Find and Teach module. 


T25. Which of these statements best describes how you created the module named in the 
previous question? 
(n = 210) 
| created a module from a template in CoreTools. 84 teachers (40.0%) 


| found and adjusted another teacher’s module from 126 teachers (60.0%) 
the LDC Library in CoreTools. 


T26. Did you teach this module in your classroom? 
(n = 210) 
Yes, | have already taught this module this year. 185 teachers (88.1%) 


No, but | plan to teach this module before the end of the 2017-18 school year. 9 teachers (4.3%) 


No, but | plan to teach this module during next school year. 13 teachers (6.2%) 


No. | do not currently have plans to teach this module in my classroom. 3 teachers (1.4%) 
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Module Peer Review 


T27. Did you attend a Peer Review/Curriculum Alignment Workshop this school year? (Y/N) 


(n = 209) 
Yes 74 teachers (35.4%) 
No 135 teachers (64.6%) 


T28. How many modules did you submit online for LDC National Peer Review during the 
current school year (2017-18)? 


| | Modules /f none, S to T29 n = 210, Mean = 0.59, Range: 0-5 


T29. How helpful did you find the National Peer Review process in improving the quality of 
your module? 


(n = 79) 
Not helpful 8 teachers (10.1%) 
A little helpful 23 teachers (29.1%) 
Moderately helpful 19 teachers (24.1%) 
Very helpful 29 teachers (36.7%) 
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7 Impact on Teacher Practice and Learning 


T30. Between the beginning and end of this year’s work with LDC, please indicate how 
much your skills have improved in the following areas: 


Not at all A little Moderately | A great deal 
Selecting focus standards for a writing assignment a Fs 22 93 83 
(n = 210) (5.7%) (10.5%) (44.3%) (39.5%) 
Creating standards-driven writing assignments 17 28 88 83 
(n = 210) (5.2%) (13.3%) (41.9%) (39.5%) 
Identifying the skills students need to develop to 12 27 97 74 
complete a writing assignment (n = 210) (5.7%) (12.9%) (46.2%) (35.2%) 
Creating daily lessons to teach the skills students need 15 40 91 64 
to complete a writing assignment (n = 210) (7.1%) (19.0%) (43.3%) (30.5%) 
Systematically collecting information on students’ 17 40 94 59 
progress (n = 210) (8.1%) (19.0%) (44.8%) (28.1%) 
Identifying patterns of student understandings or 14 28 105 63 
misconceptions (n = 210) (6.7%) (13.3%) (50.0%) (30.0%) 
Using evidence of student progress on standards to 15 28 96 ral 
modify subsequent instruction (n = 210) (7.1%) (13.3%) (45.7%) (33.8%) 


104 


T31. Please indicate the degree to which you agree or disagree with the statements below. 


Ssronely Disagree Agree Strongly 
disagree agree 
Participating in LDC raised my expectations for 7 33 113 57 
students’ writing. (n = 210) (3.3%) (15.7%) (53.8%) (27.1%) 
Using LDC modules became an important part of my 12 49 114 35 
instructional practice. (n = 210) (5.7%) (23.3%) (54.3%) (16.7%) 
Implementing LDC helped me incorporate my state’s 11 43 115 Al 
College- and Career-Ready Standards into my 5.2% 20.5% 54.8% 19.5% 
instruction. (n = 210) (5.2%) (20.5%) (54.8%) (19.5%) 
LDC helped me incorporate writing assignments into 10 29 119 BZ 
my existing curriculum. (n = 210) (4.8%) (13.8%) (56.7%) (24.8%) 
cuieiercom tana | yee . Be = 
i) i) 10) to) 
Professional Learning Community. (n = 210) (4.8%) (11.0%) (59.5%) (24.8%) 
LDC helped me improve on my teacher evaluation 16 68 55 31 
ratings. (n = 210) (7.6%) (32.4%) (45.2%) (14.8%) 
Participating in LDC helped me develop working 13 31 122 AA 
relationships with teachers in different grades and/or 6.2% 14.8% 58.1% 21.0% 
subjects. (n = 210) (6.2%) (14.8%) (58.1%) (21.0%) 
| shared my LDC work with colleagues outside of the 16 67 86 41 
LDC PLC. (n = 210) (7.6%) (31.9%) (41.0%) (19.5%) 
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Impact on Student Learning 


T32. Please indicate to what extent LDC had a positive effect on students in the following 


areas. 
Not at all A little Moderately | A great deal 
17 40 109 44 
Readi . L 
i 8.1% 19.0% 51.9% 21.0% 
(8.1%) ( ) ( ) ( ) 
10 34 109 57 
Content knowledge (n = 210) (4.8%) (16.2%) (51.9%) (27.1%) 
a vs ‘ 11 38 97 64 
Ability to complete writing assignments (n = 210) (5.2%) (18.1%) (46.2%) (30.5%) 
: er _ 12 41 101 56 
Quality of students’ writing (n = 210) (5.7%) (19.5%) (48.1%) (26.7%) 
23 46 101 40 
Coll d dy skills (n = 210 
lieEe ane concer ready chs e a) (11.0%) (21.9%) (48.1%) (19.0%) 
Capacity to analyze and understand the 12 38 101 5g 
components of a writing assignment (n = 210) (5.7%) (18.1%) (48.1%) (28.1%) 
22 46 106 36 
Speaki d listening skills (n = 210 
peaking ene isteniie hile ina (10.5%) (21.9%) (50.5%) (17.1%) 
10 43 107 50 
O Il lit f = 210 
verall literacy performance (n ) (4.8%) (20.5%) (51.0%) (23.8%) 
Performance on assessments throughout the 1S 39 111 45 
school year (n = 210) (7.1%) (18.6%) (52.9%) (21.4%) 
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Teacher Leader Support 


The following question refers to the LDC project liaison in your school. This is the teacher leading your 


Professional Learning Community work. 


T33. Please indicate the degree to which you agree or disagree with the following 


statements. 

Shovel Disagree Agree Bconely 

disagree agree 
Our school’s LDC teacher leader effectively 
supported our Professional Learning Community ° * sa = 
meetings. (n= 182) (1.6%) (2.2%) (44.0%) (52.2%) 
ee acca Soi eicia : . at a 

{0) is) {0) {0) 

leader. (n = 182) (1.6%) (0.5%) (44.5%) (53.3%) 
Our teacher leader helped teachers align LDC to 3 10 89 80 
broader school instructional goals. (n = 182) (1.6%) (5.5%) (48.9%) (44.0%) 
Our teacher leader offered useful feedback for 3 12 87 80 
the design and revision of LDC modules. (n = 182) (1.6%) (6.6%) (47.8%) (44.0%) 
Our teacher leader was effective in inviting 4 3 91 84 
teachers to join the LDC initiative. (n = 182) (2.2%) (1.6%) (50.0%) (46.2%) 


4{0)) School Administrator Support 


The following questions refer to the school administrator who oversees the LDC project at your school. 


T34. What proportion of PLC meetings focused on LDC did your school administrator attend? 


Less than one quarter of LDC PLCs 


About one quarter of LDC PLCs 
About one half of LDC PLCs 
About three quarters of LDC PLCs 


More than three quarters of LDC PLCs 
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(n = 209) 
91 teachers (43.5%) 
46 teachers (22.0%) 
30 teachers (14.4%) 
10 teachers (4.8%) 
32 teachers (15.3%) 


T35. How many times did your school administrator observe you teach an LDC mini-task 
during the current school year (2017-18)? 


(n = 209) 
0 times 84 teachers (40.2%) 
1 time 66 teachers (31.6%) 
2 times 22 teachers (10.5%) 
3 or more times 37 teachers (17.7%) 


T36. Please indicate the degree to which you agree or disagree with the following 


statements. 
My school administrator... Suroriely Disagree Agree Strongly 
disagree agree 
12 30 133 23 

had a fi derstandi f LDC. (n = 208 

edie nine Serna Oren ere (5.8%) (14.4%) (63.9%) (15.9%) 
Allocated resources such as teacher time, 
payment, administrator time, support 13 LT 124 44 
staff, sub coverage, etc., to ensure the LDC (6.3%) (13.0%) (59.6%) (21.2%) 
team could meet. (n = 208) 
encouraged teachers to participate in LDC. 9 16 124 59 
(n = 208) (4.3%) (7.7%) (59.6%) (28.4%) 
ee | ee | 
instructional priorities. (n = 208) (13.0%) (34.6%) (37.0%) (15.4%) 
et ee ele eel a . ne = 
goals. (n= 208) (5.8%) (15.4%) (60.1%) (18.8%) 
provided me with feedback about my LDC 22 57 96 33 
planning and/or instruction. (n = 208) (10.6%) (27.4%) (46.2%) (15.9%) 
made formative assessment a priority at 7 32 121 48 
my school. (n = 208) (3.4%) (15.4%) (58.2%) (23.1%) 
io ee ae ee 
‘a ai . (7.2%) (22.1%) (53.8%) (16.8%) 
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iti) Teacher Leadership Role 


T37. Please indicate the degree to which you agree or disagree with the following 
statements about your role in your school’s LDC implementation. 


strongly Disagree Agree Stronely 
disagree agree 
| was involved in setting instructional goals for the LDC 11 ys) 83 27 
work at my school. (n = 180) (6.1%) (32.8%) (46.1%) (15.0%) 
| was involved in discussions about how to expand LDC 19 62 73 26 
implementation at my school in future years. (n = 180) (10.6%) (34.4%) (40.6%) (14.4%) 
| had the opportunity to work with our LDC teacher leader 
i ; é 17 50 87 26 
and our administrator to help shape LDC implementation. 
(n= 180) (9.4%) (27.8%) (48.3%) (14.4%) 
HEHE Oe E cis ior ican anteer ial | . . 
coach, providing feedback to my peers, etc. (n = 180) (9.4%) (24.4%) (50.6%) (15.6%) 
iy Facilitators and Barriers 
T38. Please indicate the degree to which you agree or disagree with the following 
statements. 
Suonely Disagree Agree puronely 
disagree agree 
My LDC PLC was given sufficient time to meet. 12 41 118 36 
(n = 207) (5.8%) (19.8%) (57.0%) (17.4%) 
| felt adequately prepared to effectively implement 13 37 121 36 
LDC modules in my classroom. (n = 207) (6.3%) (17.9%) (58.5%) (17.4%) 
It was challenging to find content-rich reading 11 71 98 27 
materials for the LDC modules | developed. (n = 207) (5.3%) (34.3%) (47.3%) (13.0%) 
My school had adequate technology to support 30 119 50 
teachers’ use of LDC. (n = 207) (3.9%) (14.5%) (57.5%) (24.2%) 
It was easy to find and adapt LDC mini-tasks for use in 28 133 38 
my classroom. (n = 207) (3.9%) (13.5%) (64.3%) (18.4%) 
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ife}) Areas for Improvement 


There have been a number of supports for implementation of LDC in your school, including: 


e CoreTools online platform 
e LDConline courses in the “Learn” section of CoreTools 
e Virtual coaching 
o Zoom meetings, written feedback on teacher work in LDC CoreTools, emails, etc. 
e In-person coaching 


o Summer training, in-person support visits from LDC and District Lead, in-person 
professional development opportunities, etc. 


T39. What supports did you find the most useful and why? 


T40. What supports were not helpful and why? 


T41. In what ways could LDC implementation be improved in your school in the future? 
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Appendix B: 
Teacher Leader Survey and Responses 


2017-2018 
LDC Teacher Leader Survey 


LDC Participation 


TL1. Prior to the current school year (2017-18), did you have any experience with LDC? 


(n = 43) 
Yes 17 teacher leaders (39.5%) 
No y Skip toTL3 26 teacher leaders (60.5%) 


TL2. How many of the following did you teach prior to the current school year (2017-18)? 


| | LDC modules n=17, Mean = 1.94, Range: 0-5 


LDC mini-tasks, outside of modules n= 17, Mean = 2.00, Range: 0-9 
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Professional Learning Community and Teacher Collaboration 


The following questions involve the LDC Professional Learning Community (PLC) that you are leading. 


TL3. About how often did your LDC PLC meet? 


Less than once a month 


Once a month 


Every other week 


Once a week 


Twice a week or more often 


y Skip to TL5 
y Skip to TLS 
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(n = 43) 


0 teacher leaders (0.0%) 

6 teacher leaders (14.0%) 
18 teacher leaders (41.9%) 
17 teacher leaders (39.5%) 


2 teacher leaders (4.7%) 


TL4. What were the primary barriers preventing your LDC PLC from meeting weekly? 


Select all that apply. 


PLC time was not protected. 


PLC members had limited interest in attending meetings. 


School administrator did not make it a priority. 


| was unable to provide sufficient organizational support. 


Not enough teachers participated. 


PLC members had other priorities that competed with LDC 
participation. 


Other (please specify) 


(n = 24) 


6 teacher leaders (25.0%) 
5 teacher leaders (20.8%) 
4 teacher leaders (16.7%) 
3 teacher leaders (12.5%) 
3 teacher leaders (12.5%) 
17 teacher leaders (70.8%) 


O teacher leaders (0.0%) 


TLS. About how often did you have informal discussions (as opposed to scheduled 


meetings) about LDC with teachers in your LDC PLC? 


(n = 43) 
Less than once a month 3 teacher leaders (7.0%) 
Once a month 6 teacher leaders (14.0%) 
Every other week 8 teacher leaders (18.6%) 
Once a week 19 teacher leaders (44.2%) 
Twice a week or more 7 teacher leaders (16.3%) 


TL6. On average how long did your school’s LDC PLC meetings typically last? 


(n = 43) 
Less than 45 minutes 8 teacher leaders (18.6%) 
45 to 59 minutes 24 teacher leaders (55.8%) 
60 to 74 minutes 9 teacher leaders (20.9%) 
75 minutes or more 2 teacher leaders (4.7%) 
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Teacher Training and Support 


TL7. How effective was your LDC PLC in the following areas? 


Not A little Moderately Very 
effective effective effective effective 
Creating an environment in which teachers are 1 (2.3%) 2 (4.7%) 17 (39.5%) | 23 (53.5%) 


comfortable working together (n = 43) 


Fostering an environment where teachers share 0 (0.0%) 4 (9.3%) 20 (46.5%) | 19 (44.2%) 
their instructional plans with colleagues (n = 43) 


Allowing space to share student work (n = 43) 0 (0.0%) 6 (14.0%) 19 (44.2%) | 18 (41.9%) 


Helping teachers learn to improve their LDC 1 (2.3%) 8 (18.6%) | 17 (39.5%) | 17 (39.5%) 
instructional plans. (n = 43) 


TL8. How would you rate each of the following aspects of the online course material (in the 
Learn tab in LDC CoreTools) that your coach used or directed you to use? 


Poor Fair Good Excellent 
Clarity of information presented (n = 43) 0 (0.0%) 8 (18.6%) 20 (46.5%) | 15 (34.9%) 
Relevance of information presented (n = 43) 0 (0.0%) 7 (16.3%) 21 (48.8%) | 15 (34.9%) 
Ease of use (n = 43) 3 (7.0%) 7 (16.3%) 21 (48.8%) | 12 (27.9%) 
Usefulness of resource documents (e.g., LDC 
Pitfall Checklist, CCSS Mental Markers, etc.) 0 (0.0%) 9 (20.9%) 20 (46.5%) | 14 (32.6%) 
(n = 43) 
Usefulness of videos (n = 43) 3 (7.0%) 6 (14.0%) 23 (53.5%) | 11 (25.6%) 
Degree to which course material helped teachers ‘ ‘ P 
to create and/or adapt LDC modules (n = 43) 1 (2.3%) 7 (16.3%) TS (AL Sie) | 27 (99.5%) 
Opportunity to extend learning when needed ; ‘ ‘ 
ordesiredtanwal 0 (0.0%) 7 (16.3%) 19 (44.2%) | 17 (39.5%) 
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TL9a. Overall, were you able to get the feedback and support you needed from your LDC 
coach (through written feedback in LDC CoreTools, or coaching and modeling in your 
LDC PLCs) to plan, teach, reflect on, and revise LDC modules? 


(n = 43) 


Yes 43 teacher leaders (100%) 
No O teacher leaders (0.0%) 


TL9b. Did your LDC coach provide written feedback on your module(s) in LDC CoreTools in a 


timely manner? 


(n = 43) 


Yes 43 teacher leaders (100.0%) 
No O teacher leaders (0.0%) 


TL10. Outside of the PLC meetings with your LDC coach, please indicate whether you used 
each of the following types of coach support, and how helpful you found these types 


of support. 
Used 
Did not use Not A little Moderately Very 
helpful helpful helpful helpful 
Written feedback in LDC CoreTools 
Bey CUETO E Coach ine 2 (4.7%) | 0(0.0%) | 5(11.6%) | 13 (30.2%) | 23 (53.5%) 


comments areas and/or via 
the teacher work rubric) (n = 43) 


One-on-one Zoom video 
conference and/or call with your 
LDC coach (n = 43) 


3 (7.0%) 0 (0.0%) 2 (4.7%) 


10 (23.3%) 


28 (65.1%) 


Email or phone communication 
with your LDC coach (n = 43) 


2 (4.7%) 0 (0.0%) 3 (7.0%) 


12 (27.9%) 


26 (60.5%) 


Other (please specify) (n = 43) 


24 (55.8%) | 0 (0.0%) 0 (0.0%) 


5 (11.6%) 


14 (32.6%) 
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TL11. How many in-person and/or online LDC professional development offerings for school 
administrators and teacher leaders did you attend during the current school (e.g., Summer 
in-person launch days, Quarterly in-person teacher leader meetings, LDC monthly virtual 
coach meetings)? 


| | Professional development offerings n= 43, Mean = 3.98, Range: 0-15 


Support to Teacher Leader from LDC Coach 


TL12. Please indicate the degree to which you agree or disagree with the following 


statements. 


SHonely Disagree Agree Bronely N/A 

disagree agree 
| was able to reach my LDC coach if | ‘ ‘ ‘ ‘ ‘ 
had any questions aboutlDC.taeaa) 0 (0.0%) 0 (0.0%) 5 (11.6%) | 38 (88.4%) 0 (0.0%) 
LDC provided adequate technical 
support for issues with the CoreTools 0 (0.0%) 2 (4.7%) 23 (53.5%) | 18 (41.9%) 0 (0.0%) 
online platform. (n = 43) 
LDC offered sufficient professional 
development opportunities for me to r 3 r F ; 
fedd the Iniflaeethiny-schodl 0 (0.0%) 0 (0.0%) 22 (51.2%) | 21 (48.8%) 0 (0.0%) 
(n = 43) 
LDC coaches were able to connect me 
with additional resources when 0 (0.0%) 0 (0.0%) 11 (25.6%) | 31(72.1%) 1 (2.3%) 
needed. (n = 43) 
It was challenging to coordinate with 
our LDC coach on how to structure =| aava9 Gee) | a4 (25.6%) |B (ames) | e(re.e%) | 214.7%) 
Professional Learning Community 
time. (n = 43) 
When | reached out to our LDC coach, ‘ ‘ P r ‘ 
hevorche resuondedl quickly. @aasmal 0 (0.0%) 0 (0.0%) 6 (14.0%) | 36 (83.7%) 1 (2.3%) 
UPS ce civen a artewen 0 (0.0%) | 0(0.0%) | 6 (14.0%) | 36(83.7%) | 1 (2.3%) 
with.(n = 43) 
Our LDC coach was knowledgeable 
and provided high quality guidance. 0 (0.0%) 1 (2.3%) 6 (14.0%) | 35 (81.4%) 1 (2.3%) 
(n = 43) 
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Module Creation 


TL13. During the current school year (2017-18), how many LDC modules did your PLC 
individually or collaboratively adapt from existing modules (e.g., modules created in a 
prior year and/or modules from the LDC Library in CoreTools)? 


| | Adapted modules n = 43, Mean = 3.19, Range: 0-15 


TL14. During the current school year (2017-18), how many LDC modules did your PLC create 


(either individually or in a group)? Only include modules built from scratch, not those 
adapted from existing modules in the LDC library. 


| | New modules n = 43, Mean = 2.05, Range: 0-7 


TL15. How did members of your PLC collaborate to create LDC modules? Check all that 
apply. 


(n = 43) 


Modules were created by individual teachers. 


22 teacher leaders (51.2%) 


Modules were created by teams of two or more teachers. 


31 teacher leaders (72.1%) 


Modules were created by the PLC as a whole. 


9 teacher leaders (20.9%) 


Other (please specify) 


2 teacher leaders (4.7%) 
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TL16. Please indicate to what extent you were able to do each of the following when 
creating LDC modules. 


Toa 
Not at all A little bit moderate 
extent 


To a great 
extent 


Select focus standards for a writing 


ee a 1 (2.3%) 3(7.0%) | 9(20.9%) | 30 (69.8%) 


Create a standards-driven writing 


co eae 1 (2.3%) 2 (4.7%) | 10(23.3%) | 30 (69.8%) 


Select high quality, complex texts and other 
materials to engage students in deeper 1 (2.3%) 3 (7.0%) 16 (37.2%) | 23 (53.5%) 
learning (n = 43) 


Identify the skills students need to develop 
to complete a writing assignment (n = 43) 1 (2.3%) 2 (4.7%) 17 (39.5%) | 23 (53.5%) 


Create daily lessons to teach the skills a 
student needs to complete a writing 2 (4.7%) 4 (9.3%) 19 (44.2%) | 18 (41.9%) 
assignment (n = 43) 


Differentiate instruction by incorporating 
multiple ways of thinking, various levels of 2 (4.7%) 5 (11.6%) 19 (44.2%) | 17 (39.5%) 
complexity, and multiple modalities. (n = 43) 


Plan for a variety of methods to assess 
student progress (e.g., rubrics and/or mini- 1 (2.3%) 6 (14.0%) 19 (44.2%) | 17 (39.5%) 
task scoring guides) (n = 43) 


Assess the quality of writing assignments 
and/or instructional plans using Peer 
Review/Curriculum Alignment Rubric (e.g. 1 (2.3%) 2 (4.7%) 18 (41.9%) | 22 (51.2%) 
Task Pitfalls Checklist, rubric indicators) 
(n = 43) 


Make a writing assignment relevant and 


shea neioceuienteel 1 (2.3%) 2 (4.7%) | 19 (44.2%) | 21 (48.8%) 
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Impact on Student Learning 


TL17.Please indicate to what extent LDC had a positive effect on students in the following 


areas. 

Not at all Alittle Moderately | A great deal 
Reading skills (n = 43) 2 (4.7%) 8 (18.6%) 22 (51.2%) | 11 (25.6%) 
Content knowledge (n = 43) 1 (2.3%) 9 (20.9%) 20 (46.5%) | 13 (30.2%) 
Ability to complete writing assignments (n = 43) 1 (2.3%) 5 (11.6%) 18 (41.9%) | 19 (44.2%) 
Quality of students’ writing (n = 43) 1 (2.3%) 5 (11.6%) 17 (39.5%) | 20 (46.5%) 
College and career ready skills (n = 43) 3 (7.0%) 6 (14.0%) | 21(48.8%) | 13 (30.2%) 
ae heel coe the components 1 4 (2.3%) | 7 (16.3%) | 17(39.5%) | 18 (41.9%) 
Speaking and listening skills (n = 43) 4 (9.3%) 8 (18.6%) | 22 (51.2%) | 9(20.9%) 
Overall literacy performance (n = 43) 2 (4.7%) 5 (11.6%) 24 (55.8%) | 12 (27.9%) 
oe on assessments throughout the school 2 (4.7%) 5 (11.6%) 23 (53.5%) 13 (30.2%) 


School Administrator Support 


Less than one quarter of LDC PLCs 


About one quarter of LDC PLCs 


About one half of LDC PLCs 


About three quarters of LDC PLCs 
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More than three quarters of LDC PLCs 


(n = 43) 

19 teacher leaders (44.2%) 
9 teacher leaders (20.9%) 
10 teacher leaders (23.3%) 
O teacher leaders (0.0%) 

5 teacher leaders (11.6%) 


The following questions refer to the school administrator who oversees the LDC project at your school. 


TL18. What proportion of PLC meetings focused on LDC did your school administrator attend? 


TL19. Please indicate the degree to which you agree or disagree with the following 


statements. 

My school administrator... suonely Disagree Agree Strongly 
disagree agree 

had a firm understanding of LDC. (n = 43) 1 (2.3%) 7 (16.3%) | 27 (62.8%) 8 (18.6%) 
allocated resources such as teacher time, payment, 
administrator time, support staff, sub coverage, etc., 0 (0.0%) 8 (18.6%) | 26 (60.5%) 9 (20.9%) 
to ensure the LDC team could meet. (n = 43) 
encouraged teachers to participate in LDC. (n = 43) 0 (0.0%) 4 (9.3%) 26 (60.5%) | 13 (30.2%) 


expressed concerns that implementing LDC is taking 


fo} fo} fo} fo} 

time away from other instructional priorities. (n = 43) 13 (30.2%) | 12 (27.9%) | 14 (32.6%) 4 (9.3%) 
communicated how using LDC’s tools supported P r 
specific school initiatives and/or goals. (n = 43) 1 (2.3%) 5 (11.6%) 30 (69.8%) 7 (16.3%) 
provided me with feedback about my LDC planning ‘ ; ‘ ‘ 
and/orinstiuetion, (a= 43) 2 (4.7%) 12 (27.9%) | 25 (58.1%) 4 (9.3%) 
a assessment a priority at my school. 0 (0.0%) 3 (7.0%) 30 (69.8%) 10 (23.3%) 
used LDC to implement standards-driven assignments 

1 (2.3%) 7 (16.3%) 25 (58.1%) 10 (23.3%) 


within existing curriculum. (n = 43) 
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Teacher Leader Leadership Role 


TL20. Please indicate the degree to which you agree or disagree with the following 
statements about your role in leading your school’s LDC implementation. 


Sronely Disagree Agree strenely 
disagree agree 


| met regularly with my school administrator to make 


i) ie) 10) i) 
planning decisions around LDC. (n = 43) 4 (9.3%) Ta tetaee) | AP(SP ove) | Wes 20) 


| was involved in discussions about differentiating LDC 
implementation to meet teacher learning needs. 2 (4.7%) 6 (14.0%) | 23(53.5%) | 12 (27.9%) 
(n = 43) 


| was involved in discussions about how to expand LDC 


implementation at my school in future years. (n = 43) 1 (2.3%) 8 (18.6%) 23 (53.5%) | 11 (25.6%) 


My role as an LDC teacher leader allowed me to 
effectively advocate for additional resources on my 1 (2.3%) 8 (18.6%) | 26(60.5%) | 8 (18.6%) 
campus. (n = 43) 


| was involved in adjusting the problems of practice 


that my school targeted with the LDC work. (n = 43) 1 (2.3%) 4 (9.3%) 30 (69.8%) 8 (18.6%) 


| met regularly with my LDC coach to manage the LDC 


Sete ee 0(0.0%) | 4(9.3%) | 25 (58.1%) | 14 (32.6%) 


| feel that my position as an LDC teacher leader 
allowed me to build my capacity as an instructional 0 (0.0%) 4 (9.3%) 24 (55.8%) | 15 (34.9%) 
leader among my colleagues. (n = 43) 


lam confident that | can lead our LDC PLC in the 
future without the assistance of an LDC coach. 0(0.0%) | 13 (30.2%) | 21(48.8%) | 9 (20.9%) 
(n = 43) 
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Alignment 


TL21. Please indicate the degree to which you agree or disagree with the following 
statements. 


Strongly Disagree Agree Syonely 
disagree agree 


Our school connected LDC implementation to our 


specific schoolwide goals. (n = 43) 1 (2.3%) 9 (20.9%) 23 (53.5%) 10 (23.3%) 


LDC helped teachers create writing assignments to 


use within their current curricula. (n = 43) 0 (0.0%) 0 (0.0%) oh iy2.20) Lae 


LDC complemented other initiatives taking place in 


ruéehockanedat 1 (2.3%) | 3(7.0%) | 29(67.4%) | 10(23.3%) 


| view LDC as a strategy for implementing my state’s 


College- and Career-Ready Standards. (n = 43) 0 (0.0%) 5 (11.6%) 27 (62.8%) 11 (25.6%) 


The time spent implementing LDC interfered with 


other important initiatives at my school. (n = 43) 4 (9.3%) 10 (23.3%) | 23 (53.5%) 6 (14.0%) 


LDC helped prepare students in my school for 


current state assessments. (n = 43) 0 (0.0%) 7 (16.3%) 25 (58.1%) 11. (25.67%) 


It was difficult for teachers to focus on LDC because 


of other competing priorities at the school. (n = 43) 3 (7.0%) 8 (18.6%) 1D A4.2%) 13 (30.2%) 


Our instructional leaders are using LDC to implement 
standards-driven assignments within the existing 0 (0.0%) 9 (20.9%) 26 (60.5%) 8 (18.6%) 
curriculum. (n = 43) 


122 


10 


Scale-up and Sustainability 


TL22. Please indicate the degree to which you agree or disagree with the following 


statements. 
Strongly Disagree Agree euonely 
disagree agree 
| expect that most teachers participating in 
LDC this year will continue to do so next 1 (2.3%) 6 (14.0%) 28 (65.1%) 8 (18.6%) 
year. (n = 43) 
Teachers at my school who were not part of 
the LDC PLC meetings used the LDC . ‘ “ ‘ 
planning process and/or LDC CoreTools. 5 (11.67%) 23 (53.5%) 13 (30.2%) 2 (4.7%) 
(n = 43) 
As a result of LDC, new collaborations 
across grades and/or subjects were created 2 (4.7%) 11 (25.6%) 25 (58.1%) 5 (11.6%) 
or are being launched at my school. (n = 43) 
Teachers and administrators at my school 
are committed to sustaining the LDC 0 (0.0%) 11 (25.6%) 21 (48.8%) 11 (25.6%) 
initiative. (n = 43) 
| expect our LDC PLC to increase in size next 1 (2.3%) 12 (27.9%) 19 (44.2%) 11 (25.6%) 


year. (n = 43) 
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District Support 


TL23. Please indicate the degree to which you agree or disagree with the following 


statements. 
Haas pisces nore Strongly Don’t know 
gree agree 

Poe eee feet 0 (0.0%) | 2(4.7%) | 14 (32.6%) | 6 (14.0%) | 21 (48.8%) 
nie a hada firm understanding of | 9 (gqo%) | 1(2.3%) | 12 (27.9%) | 6 (14.0%) | 24 (55.8%) 
District leaders are interested in spreading 
the use of LDC to additional schools. (n = 43) 0 (0.0%) 2 (4.7%) 11 (25.6%) | 4 (9.3%) 26 (60.5%) 
District professional development efforts P ‘ ‘ ‘ P 
were aligned with the LDC initiative. (n = 43) 0 (0.0%) 1 (2.3%) 15 (34.9%) | 6 (14.0%) 21 (48.8%) 
District leaders visited my school to discuss 
ihe amplementation’et nes (n= 43) 4 (9.3%) 3 (7.0%) 13 (30.2%) | 4(9.3%) 19 (44.2%) 
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ily) Areas for Improvement 


There have been a number of supports for implementation of LDC in your school, including: 


e CoreTools online platform 
e LDConline courses in the “Learn” section of CoreTools 
e Virtual coaching 
o Zoom meetings, written feedback on teacher work in LDC CoreTools, emails, etc. 
e In-person coaching 


o Summer training, in-person support visits from LDC and District Lead, in-person 
professional development opportunities, etc. 


TL24. What supports did you find the most useful and why? 


TL25. What supports were not helpful and why? 


TL26. In what ways could LDC implementation be improved in your school in the future? 
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Appendix C: 
Administrator Survey and Responses 


2017-2018 
LDC School Administrator Survey 


LDC Participation 


A1. What is your role at the school? 


(n = 17) 
Principal 11 admins (64.7%) 
Assistant Principal 6 admins (35.3%) 
Other (please specify) 0 admin (0.0%) 


Professional Learning Community 


A2. What proportion of LDC Professional Learning Community (PLC) meetings did you 
attend during the current school year? 


(n = 17) 


Less than one quarter of LDC PLCs 7 admins (41.2%) 


About one quarter of LDC PLCs 5 admins (29.4%) 
About one half of LDC PLCs 2 admins (11.8%) 
About three quarters of LDC PLCs 2 admins (11.8%) 


More than three quarters of LDC PLCs 1 admin (5.9%) 


Training and Support 


A3. How many in-person and/or online LDC professional development offerings for school 
administrators and teacher leaders did you attend during the current school year (e.g., 
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Summer launch meetings, quarterly in-person administrator meetings, monthly virtual 


LDC leadership meetings)? 


| | LDC Professional development offerings n=16, Mean = 5.63, Range: 0-20 


A4. Please indicate the degree to which you agree or disagree with the following statements. 


Strongly Disagree Agree Stronely N/A 
disagree agree 
| was able to reach LDC staff when | had 0 (0.0%) 0 (0.0%) 4 (25.0%) 11 (68.8%) 1 (6.3%) 
questions about LDC. (n = 16) 
My school has adequate technology to 0 (0.0%) 1 (6.3%) 5 (31.3%) 10 (62.5%) 0 (0.0%) 
access LDC online resources. (n = 16) 
LDC offered sufficient professional 0 (0.0%) 0 (0.0%) 7 (43.8%) 9 (56.3%) 0 (0.0%) 
development opportunities for LDC 
teacher leaders. (n = 16) 
LDC offered sufficient professional 0 (0.0%) 0 (0.0%) 8 (50.0%) 8 (50.0%) 0 (0.0%) 
development opportunities for school 
administrators. (n = 16) 
LDC staff members were able to connect 0 (0.0%) 0 (0.0%) 7 (43.8%) 9 (56.3%) 0 (0.0%) 
me with additional resources when 
needed. (n = 16) 


Classroom Observation 


A5. On average, how many times during the school year did you observe each member of 


the LDC PLC teaching an LDC module? 


0 times GY skip toa? 


1 time 


2 times 


3 or more times 


(n = 16) 

0 admin (0.0%) 

4 admins (25.0%) 
4 admins (25.0%) 
8 admins (50.0%) 
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A6. On average, how effective were LDC modules in developing students’ literacy skills? 


(n = 16) 
Not effective 0 admin (0.0%) 
A little effective 2 admins (12.5%) 
Moderately effective 6 admins (37.5%) 
Very effective 8 admins (50.0%) 


Impact on Teacher Practice 


A7. Based on your oversight of the LDC program, please indicate on average how much the 
teaching practice of LDC PLC members improved in each of the following areas: 


modify subsequent instruction (n = 16) 


Not at all A little Moderately | A great deal 
Selecting a set of focus standards for a writing 0 (0.0%) 1 (6.3%) 6 (37.5%) 9 (56.3%) 
assignment (n = 16) 
Creating standards-driven writing assignments (n = 16) 0 (0.0%) 1 (6.3%) 5 (31.3%) 10 (62.5%) 
Identifying the skills students need to develop to 0 (0.0%) 1 (6.3%) 6 (37.5%) 9 (56.3%) 
complete a writing assignment (n = 16) 
Creating daily lessons to teach each skill a student needs 0 (0.0%) 1 (6.3%) 9 (56.3%) 6 (37.5%) 
to complete a writing assignment (n = 16) 
Systematically collecting information on students’ 0 (0.0%) 1 (6.3%) 8 (50.0%) 7 (43.8%) 
progress. (n = 16) 
Identifying patterns of student understandings or 0 (0.0%) 1 (6.3%) 7 (43.8%) 8 (50.0%) 
misconceptions (n = 16) 
Using evidence of student progress on standards to 0 (0.0%) 2 (12.5%) 6 (37.5%) 8 (50.0%) 
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Impact on Student Learning 


A8. Please indicate to what extent LDC had a positive effect on students in the following 


areas: 


school year (n = 15) 


Not at all A little Moderately | A great deal 

Reading skills (n= 15) 0 (0.0%) 1 (6.7%) 11 (73.3%) 3 (20.0%) 
Sanicienawiedaadeade) 0 (0.0%) 1 (6.7%) 8 (53.3%) 6 (40.0%) 

0, 9 0, 0, 
Ability to complete writing assignments (n = 15) oe) eee See hia 
Quality of students’ writing (n = 15) a) si ede en aan) 

0, 9 9 9 
College and career ready skills (n = 15) | ee) ee Saee) 
Capacity to analyze and understand the 0 (0.0%) 2 (13.3%) 7 (46.7%) 6 (40.0%) 
components of a writing assignment (n = 15) 

0, 9 9 0, 
Speaking and listening skills (n = 15) anal ae) anes) ers) 

0, 9 0, 0 
Overall literacy performance (n = 15) eee) aoe ia ees) 
Performance on assessments throughout the 0 (0.0%) 1 (6.7%) 10 (66.7%) 4 (26.7%) 
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7 


Administrator Leadership Role 


AY. Please indicate the degree to which you agree or disagree with the following 


statements about your role in leading LDC implementation in your school: 


SHonely Disagree Agree Stonely 
disagree agree 
| was able to shape LDC implementation at my school. (n 0 (0.0%) 1 (6.7%) 10 (66.7%) 4 (26.7%) 
= 15) 
| met regularly with the LDC teacher leader in my school 0 (0.0%) 1 (6.7%) 9 (60.0%) 5 (33.3%) 
to stay abreast of implementation progress. (n = 15) 
| was involved in discussions about differentiating LDC 0 (0.0%) 1 (6.7%) 11 (73.3%) 3 (20.0%) 
implementation to meet teacher learning needs. (n = 15) 
| led discussions about how to expand my school’s LDC 0 (0.0%) 2 (13.3%) 9 (60.0%) 4 (26.7%) 
implementation in future years. (n = 15) 
| made changes to school schedules to accommodate LDC 0 (0.0%) 1 (6.7%) 9 (60.0%) 5 (33.3%) 
professional learning time. (n = 15) 
| allocated resources such as teacher time, payment, 0 (0.0%) 1 (6.7%) 7 (46.7%) 7 (46.7%) 


administrator time, support staff, sub coverage, etc., to 
ensure the LDC team could meet. (n = 15) 
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Alignment 


A10. Please indicate the degree to which you agree or disagree with the following 


statements: 

Seenely Disagree Agree Stronely 

disagree agree 
Our school connected LDC implementation to our 0 (0.0%) 0 (0.0%) 11 (73.3%) 4 (26.7%) 
specific schoolwide goals. (n = 15) 
LDC helped teachers create writing assignments to 0 (0.0%) 0 (0.0%) 10 (66.7%) 5 (33.3%) 
use within their current curricula. (n = 15) 
LDC complemented other initiatives taking place in 0 (0.0%) 0 (0.0%) 9 (60.0%) 6 (40.0%) 
my school. (n = 15) 
| view LDC as a strategy for implementing my state’s 0 (0.0%) 0 (0.0%) 8 (53.3%) 7 (46.7%) 
College- and Career-Ready Standards. (n = 15) 
The time spent implementing LDC interfered with 2 (13.3%) 7 (46.7%) 5 (33.3%) 1 (6.7%) 
other important initiatives at my school. (n = 15) 
LDC helped prepare students in my school for current 0 (0.0%) 0 (0.0%) 12 (80.0%) 3 (20.0%) 
state assessments. (n = 15) 
It was difficult for teachers to focus on LDC because 1 (6.7%) 5 (33.3%) 9 (60.0%) 0 (0.0%) 
of other competing priorities at the school. (n = 15) 
| am using LDC to implement standards-driven 0 (0.0%) 0 (0.0%) 12 (80.0%) 3 (20.0%) 
assignments within the existing curriculum. (n = 15) 
LDC helped improve teacher evaluation ratings. 0 (0.0%) 1 (6.7%) 12 (80.0%) 2 (13.3%) 
(n = 15) 
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Scale-up and Sustainability 


A11. Please indicate the degree to which you agree or disagree with the following 


statements: 
Strongly eames Ane Strongly Don’t 
disagree agree know 
| expect that most teachers participating in LDC this 0 1 7 6 4 
year will continue to do so next year. (n = 15) (0.0%) (6.7%) (46.7%) (40.0%) (6.7%) 
Teachers at my school who were not part of the LDC 
PLC meetings used the LDC planning process and/or 7 : 2 3 
0, 9 9 9 0, 
LOC CoreTools. (n = 15) (0.0%) (53.3%) (13.3%) (13.3%) (20.0%) 
As a result of LDC, new collaborations across grades 
and/or subjects were created or are being launched ° : = . : 
0, 0, 0, 0, 9 
atmy school (eis) (0.0%) (26.7%) (60.0%) (6.7%) (6.7%) 
Teachers and administrators at my school are 0 1 10 3 dl 
committed to sustaining the LDC initiative. (n = 15) (0.0%) (6.7%) (66.7%) (20.0%) (6.7%) 
| expect our LDC PLC to increase in size 0 3 6 3 3 
next year. (n = 15) (0.0%) (20.0%) (40.0%) (20.0%) (20.0%) 
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110) District Support 


A12. Please indicate the degree to which you agree or disagree with the following 


statements: 
Strongly Bisaeree Reee Strongly Don’t 
disagree agree know 
District leaders supported the 1 (7.1%) 1 (7.1%) 4 (28.6%) 1 (7.1%) 7 (50.0%) 
implementation of LDC. (n = 14) 
District leaders had a firm understanding (752%) 1 (7.1%) 3 (21.4%) 0 (0.0%) 9 (64.3%) 


of LDC. (n = 14) 


District leaders are interested in spreading 1 (7.1%) 1 (7.1%) 2 (14.3%) 0 (0.0%) 10 (71.4%) 
the use of LDC to additional schools. (n= 


14) 

District professional development efforts 1 (7.1%) 2 (14.3%) 3 (21.4%) 0 (0.0%) 8 (57.1%) 
were aligned with the LDC initiative. (n= 

14) 


District leaders visited my school to discuss 1 (7.1%) 4 (28.6%) 4 (28.6%) 0 (0.0%) 5 (35.7%) 
the implementation of LDC. (n = 14) 
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iM) Areas for Improvement 


There have been a number of supports for implementation of LDC in your school, including: 


e CoreTools online platform 
e LDConline courses in the “Learn” section of CoreTools 
e Virtual coaching 
o Zoom meetings, written feedback on teacher work in LDC CoreTools, emails, etc. 
e  /n-person coaching 


o Summer training, in-person support visits from LDC and District Lead, in-person 
professional development opportunities, etc. 


A13. What supports did you find the most useful and why? 


A14. What supports were not helpful and why? 


A15. In what ways could LDC implementation be improved in your school in the future? 
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Appendix D: 
LDC Module Rating Dimensions 


Each module was rated on six dimensions. All of these were rated using a 5-point scale. With the first 
five, anchor points ranged from not present or realized to fully present or realized. In contrast, the 
overall dimension ranged from inadequate to advanced. 


Dimension 


1 Effective Writing Task 


Definition 

Degree to which teaching task makes effective use of the template task’s writing mode (i.e., argumentation 
or explanation); requires sustained writing and effective use of ideas and evidence to substantiate claims; 
and is feasible for most students to complete (i.e., appropriate for the grade-level and subject matter). 


Main Sources of Information 


Module Creator Handout (Task) 


e Read and evaluate the teaching task, student background/prior knowledge, and summary information. 
e Evaluate the difficulty or ease students may encounter trying to answer the question. 


e Compare module teaching task to teaching task template options. 


Anchor Points Description 


5 Fully Present or Realized The teaching task and performance expectations for the module are 
explicit and clear, require students to engage in higher-order 
thinking and writing, and are appropriate for the grade-level and 
subject matter. 


4 Sufficiently Present or Realized 


3 Moderately Present or Realized | Clear module teaching task and performance expectations are 
available, but do not require students to engage in higher-order 
thinking and writing and/or are not appropriate for the grade-level 
and subject matter. 


2 Barely Present or Realized 


1 Not Present or Realized There is minimal evidence of an effort to identify an explicit and 
clear teaching task and performance expectations that provide 
opportunities for students to think critically and are appropriate for 
the grade-level and subject matter. 
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Dimension 


2 


Alignment to CCSS, Local, State Literacy and Content Standards 


Definition 


Extent to which module addresses content essential to the discipline, as well as reading comprehension and 


writing standards informed by local and state standards. 


Main Sources of Information 


Module Creator Handout (Task) 


e Read and evaluate the standards included in the module. 


e Module should include ELA as well as subject matter CCSS/state standards. 


e Compare and contrast the standards the module includes with those that could have been included. 


e Particular attention to content standards (CCSS History/Social Studies, Science, and Technical Subjects); 


State Standards; Specific Reading, Writing, Speaking/Listening, Language Skills 


Anchor Points 


Description 


5 Fully Present or Realized 


Module specifically addresses content essential to CCSS and local or 
state standards in science or social studies, as well as reading 
comprehension and writing. All standards are well aligned to the 
topic and teaching task. 


4 Sufficiently Present or Realized 


3 Moderately Present or Realized 


Module broadly addresses content essential to CCSS and local or 
state standards in science or social studies and reading 
comprehension and writing. Standards are sufficiently aligned to the 
topic and teaching task. 


2 Barely Present or Realized 


1 Not Present or Realized 


Minimal evidence that module addresses content essential to the 
discipline and literacy standards. Standards are poorly aligned to the 
topic and teaching task. 
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Dimension 


3 


Fidelity to LDC Module Instruction 


Definition 


Degree to which module instruction, activities, and teaching task address each of the four stages of 


instructional practice (preparation for the task, reading process, transition to writing, writing process). 


Main Sources of Information 


Module Creator Handout (Instruction) and Information Sheet 


e Evaluate for distribution of activities and time spent on each of the four stages of instructional practice. 


Anchor Points 


Description 


5 Fully Present or Realized 


The module instruction, activities, and teaching task reflect 
deliberate attention and fidelity to the four discrete stages of LDC 
module instruction. Classroom materials reflect demonstrable effort 
to develop instructional scaffolding within and across each stage of 
instruction. 


4 Sufficiently Present or Realized 


3 Moderately Present or Realized 


The module instruction, activities, and teaching task reflect 
moderate attention and fidelity to the four discrete stages of LDC 
module instruction. Classroom materials reflect sufficient effort to 
develop instructional scaffolding within and across each stage of 
instruction. 


2 Barely Present or Realized 


1 Not Present or Realized 


The module instruction, activities, and teaching task reflect poor 
attention and lack of fidelity to the four discrete stages of LDC 
module instruction. Classroom materials reflect inadequate effort to 
develop instructional scaffolding within and across each stage of 
instruction. 
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Dimension 


A 


Quality Instructional Strategies 


Definition 


Degree to which the module provides clear instructional strategies aimed at helping students develop literacy 


skills and successfully complete the teaching task. In addition, the degree to which module instruction and 


activities scaffold critical thinking and performance in a way that is meaningful within the context of a given 


field or subject matter. 


Main Sources of Information 


Module Creator Handout (Instruction), Classroom Handouts, and Student Work 


e Evaluate extent to which the module activities scaffold critical thinking and student performance within 


the context of the subject matter at the core of the teaching task. 


e Evaluate extent to which instructional strategies guide student learning in literacy and ability to complete 


the teaching task. 


Anchor Points 


Description 


5 Fully Present or Realized 


Module provides clear and targeted instructional strategies and 
activities that scaffold student learning and promote critical thinking 
in social studies or science. There is explicit attention to helping 
students develop an accurate understanding of the topic and 
teaching task, and literacy skills necessary to complete the writing 
task successfully. 


4 Sufficiently Present or Realized 


3 Moderately Present or Realized 


Instructional strategies and activities are available to support 
adequate student learning and critical thinking in social studies or 
science. There is moderate attention to helping students develop an 
understanding of the topic and teaching task, and literacy skills 
necessary to complete the writing task. 


2 Barely Present or Realized 


1 Not Present or Realized 


Limited instructional strategies and activities are available to support 
student learning and critical thinking in social studies or science. 
Insufficient attention to helping students develop an understanding 
of the topic and teaching task, or literacy skills necessary to 
complete the writing task. 
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Dimension 


tS) 


Coherence and Clarity of Module 


Definition 


The degree of logical alignment found between the teaching task and the goals of the module with the 


readings, mini-tasks, and instructional strategies. 


Main Sources of Information 


Module Creator Handout (Instruction), Classroom Handouts, and Student Work 


Anchor Points 


Description 


5 Fully Present or Realized 


Strong alignment between the teaching task and goals of the 
module—including the CCSS and local and state literacy and content 
standards—with the readings, mini-tasks, student work, and 
instructional strategies. 


4 Sufficiently Present or Realized 


3 Moderately Present or Realized 


Moderate alignment between the teaching task and goals of the 
module—including the CCSS and local and state literacy and content 
standards—with the readings, mini-tasks, student work, and 
instructional strategies. 


2 Barely Present or Realized 


1 Not Present or Realized 


Poor alignment between the teaching task and goals of the 
module—including the CCSS and local and state literacy and content 
standards—with the readings, mini-tasks, student work, and 
instructional strategies. 
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Dimension 


6 Overall Impression 


Definition 
A holistic assessment of the LDC Module. 


Main Sources of Information 


Module Creator Handout, Classroom Handouts, and Student Work 


e To what extent does this module contribute to student college readiness and development of advanced 
literacy skills? 


Anchor Points 


5 Advanced LDC Module Implementation 


Proficient LDC Module Implementation 


4 
3 Adequate LDC Module Implementation 
2 


Marginal LDC Module Implementation 


- 


Inadequate LDC Module Implementation 
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Appendix E: 
Principal Interviews 


To better understand teacher retention rates of LDC schools after the 2016-2017 school 
year, we conducted an interview study of school administrators. Twenty principals and assistant 
principals volunteered to be interviewed. This section presents the methods and results of the 
principal interviews. 


Method 


Instrument. We developed a six-question interview protocol for principals and assistant 
principals. Some of the interview questions had sub-questions. Five of the questions were 
asked of all principals, regardless of their schools’ retention typology, and one question 
(question 5) differed slightly based on the school’s retention typology. See Figure E1 for the 
interview protocol. 


Question 
No. Question Given to 
1 Why did you initially participate in LDC? All 
e What were your goals when you signed on and how well have those 
goals been met thus far? 
2 What was your role in LDC implementation? All 
e Who else at the school supported LDC? 
e How much did you see of PLC meetings? 
e Was part of your role to give feedback on LDC instruction? (If 
needed: Who gave feedback on LDC instruction?) 


3 Could you tell me about outside support — from both LDC and the district — All 
and what worked or didn’t work? 
4 What were the incentives for teachers’ participation in LDC? All 


e How would you describe teacher buy-in and commitment to LDC? 
e Did teachers receive any additional pay? If so, for what? 


5 What would you say are the main factors that led to not continuing with Drop out 
LDC? 
a) Looking across the district, we noticed that some schools chose to Zero 
continue with LDC for another year and others chose to stop. What retention 


would you say are the main factors that led to continuing with LDC? 

b) We noticed that none of the year-1 teachers participated in year 2, so 
the year-2 PLC began with an entirely new set of teachers. Why do you 
think that was? 

c) Were there changes in implementation to keep this from happening 
again the following year? 
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a) Looking across the district, we noticed that some schools chose to Very Low 
continue with LDC for another year and others chose to stop. What and Low 
would you say are the main factors that led to continuing with LDC? Retention 

b) We noticed that many teachers who participated in Year 1 did not 
participate in Year 2. Why do you think that was? 


a) Looking across the district, we noticed that some schools chose to Moderate- 
continue with LDC for another year and others chose to stop. What High 
would you say are the main factors that led to continuing with LDC? Retention 


b) We noticed that many teachers who participated in Year 1 continued 

into Year 2. Why do you think that was? 
Do you have any advice for strengthening LDC or is there anything else we All 
should know about your experience with LDC? 


Figure E1. Principal and assistant principal interview protocol. 


Participants. During the 2016-2017 school year, 48 total schools participated in the i3 LDC 
program including 28 in the New York City Department of Education (NYCDOE) and 20 in the 
West Coast district which is the subject of this report. We categorized the schools into retention 
typologies based on a school’s total teacher participants in 2016-2017 and returning teachers in 
2017-2018. Using the proportions of the same teacher participating for the two years, we 
created five school retention typologies: 


drop out (the school has no teachers participating in LDC in 2017-2018); 


zero retention (no 2016-2017 teachers returning to LDC, but new teachers in the 
school are participating in 2017-2018); 


very low retention (less than 1/3 of teachers returning); 
low retention (between 1/3 and 2/3 returning teachers); and 


moderate to high retention (over 2/3 teachers returning). 


The following table show the number of schools per retention typology in NYCDOE and 
the West Coast district. 
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Table E1 
Number of Schools and Proportion of Retention Typologies for NYCDOE and West Coast district in 2016- 
2017 


Number of — Proportion of 
Number of | Proportion of West Coast total West 
NYCDOE total NYCDOE district Coast district 


Retention type schools schools schools schools 
Dropped out after 2016-2017 10 36% 6 30% 
Zero retention 1 4% 2 10% 
Very low teacher retention 7 25% 3 15% 
Low teacher retention 5 18% 2 10% 
Moderate to high teacher retention 5 18% 7 35% 
Total 28 20 


All LDC schools from the 2016-2017 cohort were contacted for a phone interview by 
email, and a total of 20 principals and assistant principals (10, or 36%, of NYCDOE schools and 
10, or 50%, of West Coast district schools) participated in the interview. Table E2 shows the 
school retention typologies that are represented in this sample. Of the principals and assistant 
principals who participated in an interview, most were from very low teacher retention schools 
(8 participants), followed by moderate to high teacher retentions schools (6 participants). We 
also had three principals or assistant principals from drop out schools participate, two principals 
from low teacher retention schools, and one principal in a zero retention school. 


Table E2 
School Retention Typologies Represented in the Study 


Nidmnbarat Proportion 
Number of Proportion eras of West 
participating of NYCDOE ee ihe Coast 
NYCDOE schools in ae district 
district F 
schools the typology schools in 
P schools 
Retention Type the typology 
Dropped out after 2016-2017 2 20% 1 17% 
Zero retention 0 0% 1 50% 
Very low teacher retention 5 71% 3 100% 
Low teacher retention 1 20% 1 50% 
Moderate to high teacher retention 2 40% 4 57% 
Total 10 10 
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Procedures. We originally contacted principals of all schools except for schools designated 
as low retention. This was done as sampling strategy for a case study methodology to 
understand reasons behind high and low retention schools. Initial emails were sent to 41 
schools in both districts (23 in NYCDOE and 18 in the West Coast district) on the week of June 
16, 2018. A minimum of three follow-up emails for non-responding administrators were sent 
between June 25 and August 6, 2018. During this window, nine NYCDOE principals or assistant 
principals participated (39% of schools in the first round) and seven principals from the West 
Coast distirct participated (39% of schools in the first round). One West Coast district principal 
contacted us to say that she did not have enough knowledge about the LDC program in the 
school because she was the new principal. 


Because of the low response rate, we decided to contact the principals in the low teacher 
retention schools (five in NYCDOE and two in the West Coast district) to increase the numbers 
of participants. The initial email for these NYCDOE principals were sent on September 20, 2018 
and, for West Coast district principals, between October 1 and October 8, 2018. Follow-up 
emails were sent to NYCDOE and West Coast district principals between the weeks of October 1 
and October 19, 2018. We also reached out to the LDC grant directors in NYCDOE and the West 
Coast district for recruitment support. Between the new round of emails and support from the 
LDC grant directors, four additional school leaders participated in interviews, bringing our total 
participants to 20 principals and assistant principals (10, or 36%, of NYCDOE schools and 10, or 
50%, of West Coast district schools). 


Interviews were scheduled at times most convenient to the principal or assistant 
principal. Interviews were conducted by two researchers following the protocol, and the 
interviews ranged from about eight minutes to 30 minutes in length. Participant consent was 
obtained prior to the interviews, and interviews were recorded. 


Analysis. Recordings of interviews were transcribed. (For one interview, due to researcher 
and equipment error, the interview was not recorded. However, notes taken by the researcher 
during the interview were used for general counts in the analysis.) Transcriptions were read 
iteratively and coded for broad themes using grounded theory (Glaser & Strauss, 1967). 
Multiple close readings of each group transcript were conducted in order to summarize the 
data and identify themes related to reasons for a school’s retention (or lack thereof) of teachers 
from year 1 to year 2 of LDC implementation. A coding scheme was developed based on 
targeted areas of interest in relation to reasons for retention. 


Results 


Qualitative results from the data are presented in five main categories: reasons for 
teachers continuing in LDC, meeting goals, administrator’s role, teacher buy-in, and outside 
support. 


Reasons for teachers continuing or leaving LDC in year 2. To understand why teachers 
remained or left LDC after year 1, interview data were coded into three main categories: 
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teacher or grade level decision, administrative decision, and mandatory participation. In 
teacher or grade level decisions, individual teachers, a grade level, or a content-area team (e.g., 
English teachers) were free to decide to return the following year. Administrative decisions 
typically involved the principal deciding on changing the grade level or content-area teachers 
who would participate in the second year of implementation (e.g., moving from second grade 
to third grade teachers; dropping math teachers but keeping science teachers in the program). 
A couple principals also decided that LDC participation would be better suited for a particular 
level of teacher. For example, a principal decided to focus year 2 teacher participation on 
“teachers that were new to the profession and teachers that were struggling instructionally;” 
therefore, only a few teachers continued from year 1 to year 2 at that site. Mandatory 
participation, which is based on an administrator’s decision, reflected schools where teachers 
were told that participation in the second year of LDC was not voluntary. Table E3 shows the 
reasons for retention by school retention typology. 


Table E3 
Teachers’ Participation in LDC in Year 2 by Retention Typology 


F Teacher/grade Administrative | Mandatory 
Retention typology 


level decision decision participation 
Dropped out after 2016-2017 1 2 0 
Zero retention 0 1 0 
Very low teacher retention 4 4 0 
Low teacher retention 1 0 1 
Moderate to high teacher retention 4 0 2 
Total 10 7 3 


For the three drop out schools in our sample, two of the schools had a change in 
leadership. One had a new principal who came in before the second year of LDC 
implementation and decided not to participate in LDC, and the other school had a new 
superintendent who did not continue partnering with LDC. For the last of the three drop out 
schools, the small group of teachers who were in the first year of implementation decided not 
to continue because of “teacher and administrator burn out.” This school had too many 
initiatives that were handled by a small number of administrative staff and the same teachers 
who were also part of the LDC implementation. 


For the one zero retention school in our sample, the principal decided to change the 
teachers for year 2. The grade level that was selected to participate in LDC the first year found 
the program successful, and so for year 2, “other teachers haven't been exposed, and we 
wanted more teachers to have access...to spread the wealth within the school,” so another 
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grade level was selected to participate. In one of the low retention schools, the principal said 
that participation in LDC was mandatory, but many of teachers who were part of year 1 
implementation left the school the following year. 


For the 10 schools that were coded as a teacher/grade level decision, half of those 
decisions were based on the grade-level or content-area team (e.g., science teachers) deciding 
to leave LDC because they started a new program or initiative the second year and did not have 
the time to participate in LDC. For the moderate to high teacher retention schools who were 
coded as teacher/grade level decision, the majority of teachers in those schools decided to stay 
in LDC the second year. 


Meeting goals. We asked participants their reasons for participating in LDC during the 
first year. The most popular reasons for participating in LDC were its alignment to standards 
and rigorous tasks. Principals and assistant principals also stated that they felt both teachers 
and students in their schools needed assignments where literacy instruction, the writing tasks, 
and the writing process were a focus. Other responses included wanting better lesson planning, 
collaboration, and content-area instruction. Interestingly, three principals said that LDC 
appealed to them as they were new to the school or position. 


After stating their reasons for participation, we asked if these goals were met after the 
first year of implementation. Table E4 shows participant responses to whether goals were met 
by school typology. The majority of principals and assistant principals responded positively: 12 
(60%) said yes, five expressed mixed feelings about accomplishing their original goals for LDC 
(25%), and three said that their goals were not met (15%). From these responses, meeting goals 
for LDC did not appear to influence retention of teachers. For example, two of the three 
dropout schools reported meeting their LDC goals for participation, yet they didn’t remain in 
the program. Teacher retention seemed to be tied more to teacher and principal decisions (see 
section above). 
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Table E4 
Principal Reponses for LDC Goals by School Typology 


LDC goals met 


School typology 


Yes No Mixed 
Dropped out after 2016-2017 2 1 0 
Zero retention 1 0 0 
Very low teacher retention 4 1 3 
Low teacher retention 1 1 0 
Moderate to high teacher retention 4 0 2 
Total 12 3 5 


Administrator’s role. The majority of principals and assistant principals in our sample 
reported having an active role in supporting LDC implementation. Sixteen (80%) attended LDC 
PLCs, and those who could not make the PLCs were able to get a summary of the meetings from 
their assistant principals or lead teachers. For those who were able to attend some PLCs, the 
amount of time they were able to sit in on the PLCs varied. Some principals were only able to 
drop in for a portion of the PLCs. For example, one principal (moderate to high retention 
school) said, “I would probably just go in the last 15 minutes. | don't think | was in a full session 
all the time. But | would touch bases and then each teacher would give me a wrap up summary 
of what they had done, what they were working on." Others were able to go to almost all PLCs. 
As the zero retention school principal said, “| was a participant, also, in the learning.” Sixteen 
participants (80%) also reported that they gave feedback on LDC instruction. However, the 
degree to which principals or assistant principals gave feedback varied. Table ES shows the 
numbers of participants who attended PLCs and provided feedback on LDC instruction. 


Table E5 
Participation of Principals and Assistant Principals in LDC Implementation by Retention Type 


Retention typology Attended PLCs Provided feedback 
Dropped out after 2016-2017 3 3 
Zero retention 1 1 
Very low teacher retention 6 6 
Low teacher retention 1 2 
Moderate to high teacher retention 5 4 


Total 16 16 
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Teacher buy-in. Most of the principals and assistant principals mentioned increasing skills 
and knowledge, collaboration, and student achievement and accessing LDC materials as 
incentives for teachers. Besides these incentives that were actually part of the LDC 
implementation, few schools were able to provide additional school-based supports. Eight 
schools (40%) in our sample reported providing teachers with more tangible incentives for their 
participation in LCD, such as additional pay and time, and four of the eight gave multiple 
incentives. Table E6 shows the number of these types teacher incentives by school typology. No 
pattern emerges that connects teacher incentives to retention typology. However, it is notable 
that the majority of schools in the sample did not provide tangible incentives (i.e., pay, extra 
time, teacher evaluation) to teachers for their participation for LDC. 


Table E6 
Teacher Incentives by School Typology 


Retention typology Additional pay Extra time Teacher evaluation 


Dropped out after 2016-2017 
Zero retention 

Very low teacher retention 
Low teacher retention 


Moderate to high teacher retention 


ule FP N RP O 
BIlP N O F Oo 
NirF oO F ODO O 


Total 


Five schools (25%) were able to pay teachers for their participation in LCD. Aside from 
pay, some principals were able to support participating teachers with extra time, such as 
adjusting the schedule to give teachers more time to meet during the week or giving teachers a 
release day. For example, the principal from the zero retention school said, “Initially, it was kind 
of an imposition of time, but when we created extra time and space for the teachers to meet, 
they were appreciative. So they had more buy-in.” Another principal (low retention school) 
said, “I make it very easy for teachers, because I've built in time in their schedule for them to 
meet for LDC. There are no discrepancies in terms of when we should meet or how can we 
meet, or what needs to be done, which is usually the issue sometimes when it comes to work 
with outside consultants or providers, right?” 


Two schools incentivized participation through teacher evaluations. One principal (very 
low retention school) postponed teacher evaluations for the year. The other principal 
(moderate to high retention school) said, “I very much tied it to teacher performance and my 
expectation of what high quality planning looks like.” For this principal, he knew LDC would be 
“transformative. | knew that if they got through it and actually did it, the carrot would reveal 
itself in the students’ performance. And | knew my teachers were committed; | knew they 
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wanted our kids to do well. They just needed to be pushed through the process to see how to 
get the kids to do well.” 


Outside support. Overwhelmingly positive responses were reported from our sample 
regarding LDC support the schools received. All principals and assistant principals mentioned 
either training or coaching that LDC provided. Two principals, both from a very low retention 
schools, also mentioned that the administrator meetings as helpful. As one principal described, 
“It gave an opportunity to speak to other people who were in the project, and it also gave an 
opportunity to actually go through all of the CoreTools that were there to support the 
teachers.” Two principals (both from New York) also found site visits to another school as 
helpful: “There are sites where the work is going on really nicely, and we were able to go over 
to those sites and see presentations from the principal and her teachers on how they have 
embedded this entire system into their daily program.” Five participants mentioned district 
support being available. 


Discussion 


Participants in this study all reported playing an active role in LDC implementation at 
their school site, and most said that their goals for participation were met. They also 
overwhelmingly responded favorably to LDC support, and district support, if provided or 
perceived, was also helpful. The principal interviews showed the diversity of reasons for why 
schools were able to retain teachers, and they also show why teachers left. The main reason for 
teacher retention was teacher and grade-level team decisions to leave or stay, followed by the 
principals’ decision to switch participation of teachers from year 1 to year 2. Teacher buy-in 
also seemed to affect teachers’ decisions for remaining or continuing. We did not find any 
consistent trends for retention typology; that is, we cannot characterize reasons for each 
retention typology and say, for example, all drop out schools had these characteristics or most 
moderate to high retention schools had other characteristics. Teacher retention in LDC from 
year 1 to year 2 was due to a myriad of factors that worked in combination at each school. 
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Appendix F: 
Module Artifact Ratings 


The following presents further details of the module analyses. First we present the 
methodology and results of the generalizability theory studies. This is followed by the 
additional tables for the descriptive analyses. 


Generalizability Study 


Generalizability theory is a statistical framework for examining multiple sources of 
potential error during the scoring process. For each grade band, we first modeled score 
variability across all six dimensions using a two-faceted design, whereby we estimated variance 
components for module by rater by dimension (t*r*d). The goal here was to separate true 
variation in the modules from other potential sources of measurement error. The main effects 
reflect true variation across modules (o2t) and error variance across raters (o2r) and 
dimensions (o2d), while the error term (o2trd,e) reflects unexplained residual error in the 
model. To disentangle the sources of potential error further, we also used a single-faceted 
design to examine potential error within the scoring of each dimension. As with the first set of 
models, the main effect reflects true variation across teachers (o2t) and error variance across 
raters (o2r). 


Elementary module results. Generalizability theory models were conducted to examine 
potential error in the scoring process for the elementary modules. Results from the two-faceted 
and one-faceted models that examine error across and within dimensions are presented in 
Tables F1 and F2. As would be the goal of any rating session, most of the variation found for the 
elementary modules was due directly to differences in the modules (48%) or to differences in 
the modules by dimension (29%). Despite this, 16% of the variation was due either directly or 
through interaction with the raters, and 5% of the variation was unexplained by the two- 
faceted model used. 
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Table F1 


Generalizability Study of the Elementary Module Ratings across Dimensions (n = 84) 


Source Var. % 
Module (o2t) 0.53 48.21 
Rater (o2r) 0.03 2.83 
Dimension (o2d) 0.01 1.27 
Module* Dimension (o2td) 0.32 28.68 
Rater* Dimension (o2rd) 0.08 7.42 
Module*Rater (o2tr) 0.07 6.18 
Error (o2trd,e) 0.06 5.41 


As previously mentioned, we also used a one-faceted design to disentangle variation in 
the ratings that was due either directly or through interaction with the dimensions (see 
Table F2). As would be desired, the greatest source of variation for each dimension was due 
directly to differences in the modules. Despite this, moderate amounts of variation could be 
attributed either directly or through interaction with the raters for Dimensions 3 through 6. For 
example, with Dimension 5, which measures coherence and clarity of the module, 10% of the 
variation was due to the raters and an additional 20% was due to an interaction between raters 
and modules. Furthermore, the greatest amount of variance due to an interaction between 
raters and modules was found for Dimensions 3 and 4 (24%, respectively), which measure 
fidelity to LDC instruction and quality instructional strategies. 


Table F2 


Generalizability Study of the Elementary Module Ratings for Each Dimension (n = 84) 


Module (o2t) Rater (o2r) Module*Rate Error 
r (o2tr) (o2trd,e) 
Dimension Var. % Var. % Var. % Var. % 
1. Effective writing task 0.76 87.39 0.02 2.30 0.09 10.31 0.00 0.00 
2. Standards alignment 0.93 86.69 0.13 12.46 0.01 0.85 0.00 0.00 
3. Fidelity to LDC instruction 0.87 68.81 0.08 6.71 0.31 24.48 0.00 0.00 


4. Quality instructional strategies 0.87 68.81 0.08 6.71 0.31 24.48 0.00 0.00 
5. Coherence/clarity of module 0.75 70.13 0.11 10.17 0.21 19.71 0.00 0.00 
6. Overall impression 0.61 62.18 0.17 16.89 0.21 20.93 0.00 0.00 
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Secondary module results. Generalizability theory models were conducted to examine potential 
error in the scoring process for the secondary modules. Tables F3 and F4 present results from the two- 
faceted and one-faceted models that examine error across and within dimensions. As would be the goal 
of any rating session, most of the variation found in the ratings was due directly to differences in the 
modules (39%) or to differences in the modules by dimension (49%). Furthermore, less than 5% of the 
variation for the secondary modules was due either directly or indirectly to the raters. What is of concern, 


though, is that 7% of the variation in ratings for the two-faceted model was unexplained. 


Table F3 


Generalizability Study of the Secondary Module Ratings across Dimensions (n = 103) 


Source Var. % 
Module (o2t) 0.48 39.15 
Rater (o2r) 0.02 1.59 
Dimension (o2d) 0.00 0.14 
Module* Dimension (o2td) 0.61 49.18 
Rater* Dimension (o2rd) 0.02 1.33 
Module*Rater (o2tr) 0.02 1.92 
Error (o2trd,e) 0.08 6.68 


We also used a one-faceted design to disentangle the variance that was due either directly or 
through interaction with the dimensions (see Table F4). As would be desired, 75% to 93% of the variance 
found for each dimension was attributable to differences in the modules. Furthermore, less than 8% of 
the variance for each dimension was due directly to the raters. The only concerns involved the moderate 
amounts of variance due to an interaction between modules and raters for Dimensions 1 and 2, which 
focus on the effective writing task and standards alignment (16% and 18%, respectively). Finally, there 


was no unexplained error variance found for any of the one-faceted models. 
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Table F4 
Generalizability Study of the Secondary Module Ratings for Each Dimension (n = 103) 


Module (o2t) Rater (o2r) Module*Rate Error 

r (o2tr) (o2trd,e) 

Dimension Var. % Var. % Var. % Var. % 
1. Effective writing task 1.20 83.86 0.01 0.40 0.23 15.74 0.00 0.00 
2. Standards alignment 0.80 74.81 0.08 7.47 0.19 17.72 0.00 0.00 
3. Fidelity to LDC instruction 1.30 92.94 0.02 1.22 0.08 5.84 0.00 0.00 
4. Quality instructional strategies 1.23. 91.75 0.03 1.91 0.08 6.34 0.00 0.00 
5. Coherence/clarity of module 1.19 91.52 0.05 4.11 0.06 4.37 0.00 0.00 
6. Overall impression 0.82 93.17 0.06 6.38 0.00 0.46 0.00 0.00 


Note. Negative estimates of variance were changed to zero in order to calculate percentages 
(see Shavelson & Webb, 1991) 


Summary. Generalizability models were fit for the overall samples for the two grade 
bands. As would be hoped for, when examining the one-faceted models, the majority of 
variation for the elementary and secondary ratings were due directly to differences in the 
modules (48%, 39%) or to differing quality in the modules across dimensions (29%, 49%). 
Likewise, results from the two-faceted models showed that the majority of variation within 
dimension was due to differences in both the elementary modules (62% to 88%) and secondary 
modules (75% to 93%). Despite this, for the elementary modules about 20% of the variation in 
the ratings was due to an interaction between the raters and modules (20% to 24%). In 
contrast, the only two dimensions in which moderate variation was found because of an 
interaction between raters and the secondary modules involved the effective writing task and 
standards alignment (16% to 18%). 


Descriptive Results 


The following section presents expanded descriptive results for both the primary and 
secondary modules. We first present background information about the modules rated. This is 
followed by descriptive statistics and percentages for the elementary modules and the 
secondary modules. Finally, we present further descriptive statistics and percentages for the 
exploratory analysis of elementary modules. 
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Table F5 


Background Variables for the Module Analysis 


Elementary Secondary Total 
Variables # % # % # % 

Cohort 

Cohort 1 Returning 10 15.38 22 28.95 32 22.70 

Cohort 1 New 1 1.54 21 27.63 22 15.60 

Cohort 2 54 83.08 33 43.42 87 61.70 
Subject 

ELA 34 52.31 44 57.89 78 55.32 

Science 9 13.85 9 11.84 18 12.77 

Social Studies 22 33.85 23 30.26 45 31.91 
Module origin 

Adapted (Other authors) 39 60.00 51 67.11 90 63.83 

Adapted (Same author) 11 16.92 9 11.84 20 14.18 

Original 15 23.08 16 21.05 31 21.99 
Author Count 

Co-authored 21 32.31 24 31.58 45 31.91 

Sole 44 67.69 52 68.42 96 68.09 
Module type 

One week, one text 3 4.62 9 11.84 12 8.51 

Regular 62 95.38 67 88.16 129 91.49 
Module components 

Student background 44 67.69 51 67.11 95 67.38 

Extension 20 30.77 25 32.89 45 31.91 

Teacher reflection 22 33.85 31 40.79 53 37.59 
Total 65 46.10 76 53.90 141 100.00 


Note. Teacher reflection counts calculated from the metadata. 
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Table F6 


Descriptive Statistics for the Elementary Modules by Content Area 


Dimension Mean Std.Dev. Median Mode Minimum Maximum 
ELA (n = 34) 
1. Effective writing task 3.76 1.21 4.00 5.00 1.00 5.00 
2. Standards alignment 3.99 0.66 4.00 4.00 3.00 5.00 
3. Fidelity to LDC instruction 3.15 1.23 3.00 4.00 1.00 5.00 
4. Quality instructional strategies 3.24 0.92 3.00 3.00 1.00 5.00 
5. Coherence/clarity of module 3.32 0.88 3.00 3.00 2.00 5.00 
6. Overall impression 3.38 0.85 3.00 3.00 2.00 5.00 
SCIENCE (n = 9) 
1. Effective writing task 3.67 1.41 4.00 4.00 1.00 5.00 
2. Standards alignment 2.44 1.51 2.00 2.00 1.00 5.00 
3. Fidelity to LDC instruction 3.67 1.22 4.00 4.00 1.00 5.00 
4. Quality instructional strategies 2.78 1.20 3.00 3.00 1.00 4.00 
5. Coherence/clarity of module 2.33 0.87 3.00 3.00 1.00 3.00 
6. Overall impression 2.89 0.93 3.00 3.00 1.00 4.00 
SOCIAL STUDIES (n = 22) 
1. Effective writing task 4.36 1.00 4.00 5.00 1.00 5.00 
2. Standards alignment 4.09 1.11 4.00 5.00 1.00 5.00 
3. Fidelity to LDC instruction 4.14 1.13 4.50 5.00 1.00 5.00 
4. Quality instructional strategies 3.45 1.37 4.00 4.00 1.00 5.00 
5. Coherence/clarity of module 3.64 1.18 4.00 4.00 1.00 5.00 
6. Overall impression 3.86 0.94 4.00 4.00 2.00 5.00 
OVERALL (n = 65) 
1. Effective writing task 3.95 1.19 4.00 5.00 1.00 5.00 
2. Standards alignment 3.81 1.10 4.00 4.00 1.00 5.00 
3. Fidelity to LDC instruction 3.55 1.26 4.00 4.00 1.00 5.00 
4. Quality instructional strategies 3.25 1.13 3.00 3.00 1.00 5.00 
5. Coherence/clarity of module 3.29 1.06 3.00 3.00 1.00 5.00 
6. Overall impression 3.48 0.94 3.00 3.00 1.00 5.00 
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Table F7 


Distribution (Percentage) of Ratings for the Elementary Modules by Content Area 


Dimension 1 2 3 4 5 
ELA (n = 43) 
1. Effective writing task 2.94 17.65 14.71 29.41 35.29 
2. Standards alignment 0.00 0.00 20.59 55.88 20.59 
3. Fidelity to LDC instruction 11.76 20.59 20.59 35.29 11.76 
4. Quality instructional strategies 2.94 14.71 47.06 26.47 8.82 
5. Coherence/clarity of module 0.00 17.65 41.18 32.35 8.82 
6. Overall impression 0.00 14.71 41.18 35.29 8.82 
SCIENCE (n = 9) 
1. Effective writing task 11.11 11.11 11.11 33.33 33.33 
2. Standards alignment 22.22 55.56 0.00 0.00 22.22 
3. Fidelity to LDC instruction 11.11 0.00 22.22 44.44 22.22 
4. Quality instructional strategies 22.22 11.11 33.33 33.33 0.00 
5. Coherence/clarity of module 22.22 22.22 55.56 0.00 0.00 
6. Overall impression 11.11 11.11 55.56 22.22 0.00 
SOCIAL STUDIES (n = 22) 
1. Effective writing task 4.55 0.00 9.09 27.27 59.09 
2. Standards alignment 4.55 4.55 13.64 31.82 45.45 
3. Fidelity to LDC instruction 4.55 4.55 13.64 27.27 50.00 
4. Quality instructional strategies 13.64 9.09 22.73 27.27 27.27 
5. Coherence/clarity of module 9.09 4.55 22.73 40.91 22.73 
6. Overall impression 0.00 9.09 22.73 40.91 27.27 
OVERALL (n = 65) 
1. Effective writing task 4.62 10.77 12.31 19.23 43.08 
2. Standards alignment 4.62 9.23 15.38 40.00 29.23 
3. Fidelity to LDC instruction 9.23 12.31 18.46 33.85 26.15 
4. Quality instructional strategies 9.23 12.31 36.92 27.69 13.85 
5. Coherence/clarity of module 6.15 13.85 36.92 30.77 12.31 
6. Overall impression 1.54 12.31 36.92 35.38 13.85 
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Table F8 


Descriptive Statistics for the Elementary Modules by Cohort 


Dimension Mean Std.Dev. Median Mode Minimum Maximum 


COHORT 1 RETURNING (n = 10) 


1. Effective writing task 4.10 0.99 4.00 4.00 2.00 5.00 
2. Standards alignment 4.10 0.74 4.00 4.00 3.00 5.00 
3. Fidelity to LDC instruction 3.80 0.92 4.00 4.00 2.00 5.00 
4. Quality instructional strategies 3.00 1.05 3.00 3.00 1.00 5.00 
5. Coherence/clarity of module 3.10 0.99 3.00 3.00 1.00 4.00 
6. Overall impression 3.50 0.85 3.50 3.00 2.00 5.00 
COHORT 1 NEW (n= 1) 

1. Effective writing task 5.00 5.00 5.00 5.00 5.00 5.00 
2. Standards alignment 5.00 5.00 5.00 5.00 5.00 5.00 
3. Fidelity to LDC instruction 5.00 5.00 5.00 5.00 5.00 5.00 
4. Quality instructional strategies 5.00 5.00 5.00 5.00 5.00 5.00 
5. Coherence/clarity of module 5.00 5.00 5.00 5.00 5.00 5.00 
6. Overall impression 5.00 5.00 5.00 5.00 5.00 5.00 
COHORT 2 (n = 54) 

1. Effective writing task 3.91 1.23 4.00 5.00 1.00 5.00 
2. Standards alignment 3.73 1.15 4.00 4.00 1.00 5.00 
3. Fidelity to LDC instruction 3.48 1.31 4.00 4.00 1.00 5.00 
4. Quality instructional strategies 3.26 1.14 3.00 3.00 1.00 5.00 
5. Coherence/clarity of module 3.30 1.06 3.00 3.00 1.00 5.00 
6. Overall impression 3.44 0.95 3.00 3.00 1.00 5.00 
OVERALL (n = 65) 

1. Effective writing task 3.95 1.19 4.00 5.00 1.00 5.00 
2. Standards alignment 3.81 1.10 4.00 4.00 1.00 5.00 
3. Fidelity to LDC instruction 3.55 1.26 4.00 4.00 1.00 5.00 
4. Quality instructional strategies 3.25 1.13 3.00 3.00 1.00 5.00 
5. Coherence/clarity of module 3.29 1.06 3.00 3.00 1.00 5.00 
6. Overall impression 3.48 0.94 3.00 3.00 1.00 5.00 


156 


Table F9 


Distribution (Percentage) of Ratings for the Elementary Modules by Cohort 


Dimension 1 2 3 4 5 
COHORT 1 RETURNING (n = 10) 
1. Effective writing task 0.00 10.00 10.00 40.00 40.00 
2. Standards alignment 0.00 0.00 20.00 50.00 30.00 
3. Fidelity to LDC instruction 0.00 10.00 20.00 50.00 20.00 
4. Quality instructional strategies 10.00 10.00 60.00 10.00 10.00 
5. Coherence/clarity of module 10.00 10.00 40.00 40.00 0.00 
6. Overall impression 0.00 10.00 40.00 40.00 10.00 
COHORT 1 NEW (n= 11) 
1. Effective writing task 0.00 0.00 0.00 0.00 100.00 
2. Standards alignment 0.00 0.00 0.00 0.00 100.00 
3. Fidelity to LDC instruction 0.00 0.00 0.00 0.00 100.00 
4. Quality instructional strategies 0.00 0.00 0.00 0.00 100.00 
5. Coherence/clarity of module 0.00 0.00 0.00 0.00 100.00 
6. Overall impression 0.00 0.00 0.00 0.00 100.00 
COHORT 2 (n = 54) 
1. Effective writing task 5.56 11.11 12.96 27.78 42.59 
2. Standards alignment 5.56 11.11 14.81 1.85 38.89 
3. Fidelity to LDC instruction 11.11 12.96 18.52 31.48 25.93 
4. Quality instructional strategies 9.26 12.96 33.33 31.48 12.96 
5. Coherence/clarity of module 5.56 14.81 37.04 29.63 12.96 
6. Overall impression 1.85 12.96 37.04 35.19 12.96 
OVERALL (n = 65) 
1. Effective writing task 4.62 10.77 12.31 19.23 43.08 
2. Standards alignment 4.62 9.23 15.38 40.00 29.23 
3. Fidelity to LDC instruction 9.23 12.31 18.46 33.85 26.15 
4. Quality instructional strategies 9.23 12.31 36.92 27.69 13.85 
5. Coherence/clarity of module 6.15 13.85 36.92 30.77 12.31 
6. Overall impression 1.54 12.31 36.92 35.38 13.85 
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Table F10 


Descriptive Statistics for the Secondary Modules by Content Area 


Dimension Mean Std.Dev. Median Mode Minimum Maximum 
ELA (n = 44) 
1. Effective writing task 3.16 1.03 3.00 3.00 1.00 5.00 
2. Standards alignment 3.18 0.97 3.00 3.00 1.00 5.00 
3. Fidelity to LDC instruction 3.18 1.32 3.00 3.00 1.00 5.00 
4. Quality instructional strategies 3.20 1.11 3.00 3.00 1.00 5.00 
5. Coherence/clarity of module 3.09 1.05 3.00 4.00 1.00 5.00 
6. Overall impression 3.09 0.83 3.00 3.00 1.00 5.00 
SCIENCE (n = 9) 
1. Effective writing task 4.00 1.00 4.00 4.00 2.00 5.00 
2. Standards alignment 2.56 1.01 2.00 2.00 2.00 5.00 
3. Fidelity to LDC instruction 3.22 1.30 3.00 3.00 1.00 5.00 
4. Quality instructional strategies 3.11 1.36 3.00 3.00 1.00 5.00 
5. Coherence/clarity of module 2.89 0.93 3.00 3.00 1.00 4.00 
6. Overall impression 3.00 1.00 3.00 3.00 1.00 4.00 
SOCIAL STUDIES (n = 23) 
1. Effective writing task 3.91 1.20 4.00 5.00 1.00 5.00 
2. Standards alignment 3.78 0.74 4.00 4.00 2.00 5.00 
3. Fidelity to LDC instruction 3.83 1.07 4.00 4.00 1.00 5.00 
4. Quality instructional strategies 3.61 1.53 4.00 4.00 0.00 5.00 
5. Coherence/clarity of module 3.61 1.44 4.00 5.00 0.00 5.00 
6. Overall impression 3.87 1.01 4.00 5.00 2.00 5.00 
OVERALL (n = 76) 
1. Effective writing task 3.49 1.14 3.00 3.00 1.00 5.00 
2. Standards alignment 3.29 0.98 3.00 3.00 1.00 5.00 
3. Fidelity to LDC instruction 3.38 1.26 3.00 3.00 1.00 5.00 
4. Quality instructional strategies 3.32 1.28 3.50 4.00 0.00 5.00 
5. Coherence/clarity of module 3.22 1.18 3.00 4.00 0.00 5.00 
6. Overall impression 3.32 0.97 3.00 3.00 1.00 5.00 
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Table F11 


Distribution (Percentage) of Ratings for the Secondary Modules by Content Area 


Dimension 1 2 3 4 5 
ELA (n = 44) 
1. Effective writing task 9.09 9.09 47.73 25.00 9.09 
2. Standards alignment 6.82 11.36 45.45 29.55 6.82 
3. Fidelity to LDC instruction 15.91 9.09 36.36 18.18 20.45 
4. Quality instructional strategies 11.36 9.09 36.36 34.09 9.09 
5. Coherence/clarity of module 9.09 18.18 31.82 36.36 4.55 
6. Overall impression 6.82 6.82 59.09 25.00 2.27 
SCIENCE (n = 9) 
1. Effective writing task 0.00 11.11 11.11 44.44 33.33 
2. Standards alignment 0.00 66.67 22.22 0.00 11.11 
3. Fidelity to LDC instruction 11.11 11.11 44.44 11.11 22.22 
4. Quality instructional strategies 11.11 22.22 33.33 11.11 22.22 
5. Coherence/clarity of module 11.11 11.11 55.56 22.22 0.00 
6. Overall impression 11.11 11.11 44.44 33.33 0.00 
SOCIAL STUDIES (n = 23) 
1. Effective writing task 4.35 4.35 34.78 8.70 47.83 
2. Standards alignment 0.00 4.35 26.09 56.52 13.04 
3. Fidelity to LDC instruction 4.35 4.35 26.09 34.78 30.43 
4. Quality instructional strategies 4.35 4.35 13.04 39.13 30.43 
5. Coherence/clarity of module 4.35 13.04 17.39 26.09 34.78 
6. Overall impression 0.00 8.70 30.43 26.09 34.78 
OVERALL (n = 76) 
1. Effective writing task 6.58 7.89 39.47 22.37 23.68 
2. Standards alignment 3.95 15.79 36.84 34.21 9.21 
3. Fidelity to LDC instruction 11.84 7.89 34.21 22.37 23.68 
4. Quality instructional strategies 9.21 9.21 28.95 32.89 17.11 
5. Coherence/clarity of module 7.89 15.79 30.26 31.58 13.16 
6. Overall impression 5.26 7.89 48.68 26.32 11.84 


159 


Table F12 


Descriptive Statistics for the Secondary Modules by Cohort 


Dimension Mean Std.Dev. Median Mode Minimum Maximum 


COHORT 1 RETURNING (n = 22) 


1. Effective writing task 3.41 1.26 3.00 3.00 1.00 5.00 
2. Standards alignment 2.95 1.05 3.00 3.00 1.00 5.00 
3. Fidelity to LDC instruction 3.45 1.26 3.00 3.00 1.00 5.00 
4. Quality instructional strategies 3.45 0.96 4.00 4.00 1.00 5.00 
5. Coherence/clarity of module 3.45 0.86 3.50 4.00 2.00 5.00 
6. Overall impression 3.41 0.73 3.00 3.00 2.00 5.00 
COHORT 1 NEW (n = 21) 

1. Effective writing task 3.33 1.11 3.00 3.00 1.00 5.00 
2. Standards alignment 3.43 1.08 4.00 4.00 1.00 5.00 
3. Fidelity to LDC instruction 3.19 1.47 3.00 4.00 1.00 5.00 
4. Quality instructional strategies 2.90 1.61 3.00 5.00 0.00 5.00 
5. Coherence/clarity of module 2.67 1.39 3.00 3.00 1.00 5.00 
6. Overall impression 3.05 1.24 3.00 3.00 1.00 5.00 
COHORT 2 (n = 33) 

1. Effective writing task 3.64 1.08 4.00 3.00 1.00 5.00 
2. Standards alignment 3.42 0.83 4.00 4.00 2.00 5.00 
3. Fidelity to LDC instruction 3.45 1.15 3.00 3.00 1.00 5.00 
4. Quality instructional strategies 3.48 1.20 4.00 4.00 0.00 5.00 
5. Coherence/clarity of module 3.42 1.15 4.00 4.00 0.00 5.00 
6. Overall impression 3.42 0.90 3.00 3.00 1.00 5.00 
OVERALL (n = 76) 

1. Effective writing task 3.49 1.14 3.00 3.00 1.00 5.00 
2. Standards alignment 3.29 0.98 3.00 3.00 1.00 5.00 
3. Fidelity to LDC instruction 3.38 1.26 3.00 3.00 1.00 5.00 
4. Quality instructional strategies 3.32 1.28 3.50 4.00 0.00 5.00 
5. Coherence/clarity of module 3.22 1.18 3.00 4.00 0.00 5.00 
6. Overall impression 3.32 0.97 3.00 3.00 1.00 5.00 
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Table F13 


Distribution (Percentage) of Ratings for the Secondary Modules by Cohort 


Dimension 1 2 3 4 5 
COHORT 1 RETURNING (n = 22) 
1. Effective writing task 9.09 9.09 40.91 13.64 27.27 
2. Standards alignment 9.09 18.18 50.00 13.64 9.09 
3. Fidelity to LDC instruction 9.09 9.09 36.36 18.18 27.27 
4. Quality instructional strategies 4.55 9.09 31.82 45.45 9.09 
5. Coherence/clarity of module 0.00 13.64 36.36 40.91 9.09 
6. Overall impression 0.00 4.55 59.09 27.27 9.09 
COHORT 1 NEW (n = 21) 
1. Effective writing task 4.76 14.29 42.86 19.05 19.05 
2. Standards alignment 4.76 14.29 28.57 38.10 14.29 
3. Fidelity to LDC instruction 19.05 14.29 19.05 23.81 23.81 
4. Quality instructional strategies 19.05 19.05 19.05 14.29 23.81 
5. Coherence/clarity of module 28.57 14.29 33.33 9.52 14.29 
6. Overall impression 14.29 14.29 38.10 19.05 14.29 
COHORT 2 (n = 33) 
1. Effective writing task 6.06 3.03 36.36 30.30 24.24 
2. Standards alignment 0.00 15.15 33.33 45.45 6.06 
3. Fidelity to LDC instruction 9.09 3.03 42.42 24.24 21.21 
4. Quality instructional strategies 6.06 3.03 33.33 36.36 18.18 
5. Coherence/clarity of module 0.00 18.18 24.24 39.39 15.15 
6. Overall impression 3.03 6.06 48.48 30.30 12.12 
OVERALL (n = 76) 
1. Effective writing task 6.58 7.89 39.47 22.37 23.68 
2. Standards alignment 3.95 15.79 36.84 34.21 9.21 
3. Fidelity to LDC instruction 11.84 7.89 34.21 22.37 23.68 
4. Quality instructional strategies 9.21 9.21 28.95 32.89 17.11 
5. Coherence/clarity of module 7.89 15.79 30.26 31.58 13.16 
6. Overall impression 5.26 7.89 48.68 26.32 11.84 
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Table F14 


Descriptive Statistics for the Exploratory Analysis of Elementary Modules 


Dimension Mean Std.Dev. Median Mode Minimum Maximum 


2016-2017 (n= 5) 


1. Effective writing task 4.00 1.73 5.00 5.00 1.00 5.00 
2. Standards alignment 3.40 1.52 4.00 4.00 1.00 5.00 
3. Fidelity to LDC instruction 2.60 1.14 3.00 3.00 1.00 4.00 
4. Quality instructional strategies 3.40 1.34 4.00 4.00 2.00 5.00 
5. Coherence/clarity of module 3.20 1.92 4.00 4.00 0.00 5.00 
6. Overall impression 3.20 1.48 3.00 3.00 1.00 5.00 
2017—2018 (n= 5) 

1. Effective writing task 4.20 0.84 4.00 5.00 3.00 5.00 
2. Standards alignment 4.00 0.71 4.00 4.00 3.00 5.00 
3. Fidelity to LDC instruction 4.00 0.71 4.00 4.00 3.00 5.00 
4. Quality instructional strategies 2.80 1.10 3.00 3.00 1.00 4.00 
5. Coherence/clarity of module 3.00 1.22 3.00 4.00 1.00 4.00 
6. Overall impression 3.40 0.89 4.00 4.00 2.00 4.00 
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Table F15 


Distribution (Percentage) of Ratings for the Exploratory Analysis of Elementary Modules 


Dimension 1 2 3 4 5 

2016—2017 (n= 5) 

1. Effective writing task 20.00 0.00 0.00 20.00 60.00 
2. Standards alignment 20.00 0.00 20.00 40.00 20.00 
3. Fidelity to LDC instruction 20.00 20.00 40.00 20.00 0.00 
4. Quality instructional strategies 0.00 40.00 0.00 40.00 20.00 
5. Coherence/clarity of module 20.00 0.00 20.00 40.00 20.00 
6. Overall impression 0.00 0.00 40.00 20.00 20.00 
2017—2018 (n = 5) 

1. Effective writing task 0.00 0.00 20.00 40.00 40.00 
2. Standards alignment 0.00 0.00 20.00 60.00 20.00 
3. Fidelity to LDC instruction 0.00 0.00 20.00 60.00 20.00 
4. Quality instructional strategies 20.00 0.00 60.00 20.00 0.00 
5. Coherence/clarity of module 20.00 0.00 40.00 40.00 0.00 
6. Overall impression 0.00 20.00 20.00 60.00 0.00 
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Table F16 


Descriptive Statistics for the Exploratory Analysis of Secondary Modules 


Dimension Mean Std.Dev. Median Mode Minimum Maximum 


2016-2017 (n = 14) 


1. Effective writing task 3.29 1.20 3.50 4.00 1.00 5.00 
2. Standards alignment 3.14 1.17 3.00 3.00 1.00 5.00 
3. Fidelity to LDC instruction 3.14 1.29 3.00 4.00 1.00 5.00 
4. Quality instructional strategies 3.00 1.24 3.00 3.00 1.00 5.00 
5. Coherence/clarity of module 2.93 1.38 3.00 3.00 1.00 5.00 
6. Overall impression 3.07 1.14 3.00 3.00 1.00 5.00 
2017—2018 (n = 14) 

1. Effective writing task 3.57 1.22 3.00 3.00 2.00 5.00 
2. Standards alignment 2.93 1.14 3.00 3.00 1.00 5.00 
3. Fidelity to LDC instruction 3.29 1.59 3.00 5.00 1.00 5.00 
4. Quality instructional strategies 3.93 0.62 4.00 4.00 3.00 5.00 
5. Coherence/clarity of module 3.79 0.80 4.00 3.00 3.00 5.00 
6. Overall impression 3.64 0.84 3.00 3.00 3.00 5.00 
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Table F17 


Distribution (Percentage) of Ratings for the Exploratory Analysis of Secondary Modules 


Dimension 1 2 3 4 5 

2016-2017 (n = 14) 

1. Effective writing task 7.14 21.43 21.43 35.71 14.29 
2. Standards alignment 7.14 21.43 35.71 21.43 14.29 
3. Fidelity to LDC instruction 14.29 14.29 28.57 28.57 14.29 
4. Quality instructional strategies 14.29 14.29 42.86 14.29 14.29 
5. Coherence/clarity of module 21.43 14.29 28.57 21.43 14.29 
6. Overall impression 14.29 21.43 42.86 14.29 14.29 
2017—2018 (n = 14) 

1. Effective writing task 0.00 21.43 35.71 7.14 35.71 
2. Standards alignment 14.29 14.29 42.86 21.43 7.14 
3. Fidelity to LDC instruction 21.43 7.14 28.57 7.14 35.71 
4. Quality instructional strategies 0.00 0.00 21.43 64.29 14.29 
5. Coherence/clarity of module 0.00 0.00 42.86 35.71 21.43 
6. Overall impression 0.00 0.00 57.14 21.43 21.43 
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Appendix G: 
Outcome Analysis Methodology 


Analysis Model Specification 


The specification outlined below is for middle school analyses. The elementary model is 
identical with two exceptions. A variable representing the average number of weeks in which a 
student was exposed to core class content was included in the middle school models to capture 
variation in that variable due to students taking extra core science content. Additionally, the 
middle school models include two dummy variables (one for grade 7 and one for grade 8), while 
the elementary school model only required one dummy variable (grade 5). 


For our outcome analyses, we used a threshold of p < .05 to determine whether there was 
a Statistically significant impact of LDC on ELA achievement. In addition to the LDC treatment 
indicator, a teacher effect for years of experience was included, as well as an aggregate 
indicator measuring the mean prior performance of each student’s classroom peers. The fixed 
effects also included student characteristics to identify the matched comparison sample of 
students, such as baseline achievement, socioeconomic status, demographics, language 
proficiency, grade level, and participation in special education. 


The three-level MMMC model was used to estimate the impacts of the LDC intervention 
on student learning. This same analytic model will be used to estimate impacts in future years. 
The general specification for the middle school level MMMC model is shown in the following 
equation using similar notation proposed by Browne et al. (2001, equation 6) and applied in 
Tranmer, Steel, and Browne (2014, equation 3). 


eee | (3) (2) 
Vi = XB + Uschoo(i) > Wi ju +e; 


j€Teacher (i) 


i = 1,...,n Teacher(i) C (1,...,J) 


(3) (2) 
Uschool(i) ~ N(0, o%¢2)), Uj = N(0, O(2))) ~ N(0,02) 


In this model y; is the student achievement score response, Xiis a vector of the fixed 
covariates and f is the vector of the corresponding fixed effects. School(i) is the school which 


student i attends, thus the term yu? represents the random effects for that level of 
(2) | (2) 


School(i) 
classification. Within the term >) jereacner(i) Wijuj , u;~ Is the set of j random effects for the 
teachers included in the selected dataset, and w, ; is the weight which sums to 1 for each 
student applied in proportion to the instruction time assigned with each teacher. The following 


presents an example of the full model middle school specification. 
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achievement; = By + LDC; * 6, + priorELA; * B2 + priorMath, * B, + Female; * 6, + LEP; 
* Bs +SES; * B, + Black; * B; + Hispanic; * Bg + Asian; * Big + OtherEth; 
* By, + SPED; * B12 + Grade7; * B,3 + Grade8; * By4 
+priorClassELA, * B,5 + TeachExp; * B16 


+ ContentExpWeeks; * By7 + We eit > wi ju” +e; 


j 
j€Teacher(i) 


(3) (2) 
Uschool(i) ~ N(0, o%¢3)), Uj ~ N(O, Ox(2))s ei ~ N(0,02) 


In this model, achievement; is the standardized ELA outcome score for student i; In the 
dosage-dependent model LDC; is the proportion of core class instruction time taken with an 
LDC teacher (ranges 0-1); In the dosage-independent model LDC; is coded as zero for 
comparison students and as 1 for students receiving any level of LDC teacher exposure. 


Female;, 
LEP;, Black;, Hispanic; Asian;, OtherEth,, SPED,, SES;, Grade7;, Grade8; are student 
demographic indicators coded 1 if the status is present and 0 if absent; white students serve as 
the reference group for the race/ethnicity variables, and grade level 6 is the reference group for 
grade level; 


priorELA, , and priorMath, are standardized student achievement scores from the prior 
year; 


priorClassELA; is the aggregated mean of the prior ELA scores for all the core class 
peers of student i; 


TeachExp;, is the aggregated percentage of the years of teaching experience less than 3 
for those teachers which student i was exposed to in her core classes; 


ContentExpWeeks, is the average number of weeks in which student i was exposed to in 
her core classes; 


B, is the impact of LDC, the treatment; 

B12 is the average difference between Cohort 1 and Cohort 2; 
Bz & B3 are the effects of the prior score covariates; 

By... By4 are the effects of the demographic covariates; 


Bi5& By, are the effects of the aggregated class level covariates 


y® 


(2) 
School(i’ Uj» ei are the error components at the school, teacher, and student level 


respectively assumed to all have mean O and variance, Oun(2) Ou2(3) og respectively. 
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Student/Teacher Course Exposure Weighting 


Tables G1 and G2 demonstrate how the process of calculating general MMMC teacher 
weights and LDC treatment weights (for the dosage modeling approach) was conducted 
respectively for elementary and middle school. In elementary school, in the event that a 
student was exposed to more than one teacher, each content area was given equal weight in 
distributing teacher/student exposure. For example, if a student was enrolled for both ELA and 
social studies/history under one teacher, then that teacher was coded as .67 for having 
contributed to two thirds of the students’ core curriculum exposure. If the same student 
enrolled in science with a different teacher than the one who was linked to their course marks 
in ELA and social studies/history, then that science teacher would have been coded as .33 and 
all other teachers in the sample would have been coded as zero. This would then result in the 
student’s exposure adding to a unity (1). 


Table G1 
Example of Elementary School Student/Teacher Weighting Based on Course Links 


Example student/teacher weighting for use in 


MMMC Example LDC 
Marking (Weight=Marking Period/Total Marking treatment 
Period Period3) dosage weight 
One Student enrolled Weight=.333 


with an intervention Teacher: 
Weight=(1/3)=0.333 

Two Student enrolled Weight=.333 
with an intervention Teacher: 
Weight=(1/3)=0.333 

Three Student enrolled Weight=.0 
with a non-intervention teacher: 
Weight=(1/3)=0.333 

Total Unity: for every student the student/teacher Treatment 
weights sum to 1 Weight=.667 


Note. In the selected samples for these analyses, core content courses within each 
marking period were taught by a single teacher. 
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Table G2 
Example of Middle School Student/Teacher Weighting Based on Course Mark Links 


Example student/teacher weighting for use in Example LDC 


Core content MMMC treatment 
area Course name (Weight=Subject Days/Total Days) dosage weight 
ELA English 7A & 7B Student enrolled two terms of core ELA (grade Weight=.400 


7) with an intervention teacher: 
Weight=2/5=0.400 


Social studies Social Studies WHG: Student enrolled two terms of core social Weight=.0 
ANC CIV &B studies (grade 7) with a non-intervention 
teacher: 


Weight=2/5=0.400 


Science Science 7 Student enrolled one terms of core science Weight=.0 
(grade 7) with a non-intervention teacher: 


Weight=1/5=0.200 


Total Unity: for every student the student/teacher Treatment 
weights sum to 1 Weight=.400 


Note. “Days” refers to core content enrolled days preceding the NYS assessment date. 


In middle school, students’ exposure to teachers at the course level in the three core 
content areas was coded in the same manner as in the elementary grades based on enrolled 
time preceding the assessment period. A difference in our middle school coding process was 
that we did not force each core content area into equal weighting. Instead each core content 
area exposure contributed to a core content area total sum which formed the basis from which 
the weights were proportioned. 


Most commonly a student had equivalent days of core instruction exposure in each of the 
three content areas (often 215 days in each content area). In that scenario, if a student had 
exposure to three different teachers, then each teacher would contribute one third (.33) of the 
overall core curriculum exposure and all other teachers in the sample would be coded as zero. 
However, in addition to the typical core science course, extra core science courses were also 
included in the LDC analysis (for example a Grade 8 student taking biology), which made it 
possible then for a student to accumulate more units in science than in the other two content 
areas. 


The weighting in middle school was always distributed as a proportion of the total 
exposure days in the three content areas. Therefore, if a student accumulated 300 science days 
(across two courses), 200 social studies days, and 200 ELA days, the base number of instruction 
days would be 700 days. If, using that same scenario, the same teacher taught both the typical 
core and biology courses then that teacher would contribute three-sevenths (.43) of the overall 
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core curriculum exposure with the social studies and science teachers contributing two- 
sevenths (.285) each, again resulting in the student’s exposure adding to a unity (1). 


Calculation of Effect Size 


We calculated student-level effect sizes according to the WWC 3.0 criteria. Specifically, for 
the impact analysis with treatment status as a dichotomous variable, we calculated Hedges’ g, 
the difference in adjusted mean outcomes for the groups divided by the unadjusted pooled 
within-group standard deviation of the outcome measure in the sample, for all outcomes. The 
difference in adjusted mean outcomes is estimated by £, in the models defined above, as 
outlined in the WWC standards handbook for computing effect sizes in multi-level frameworks. 
Specifically, 


wy 

QO 

(nj - 1)s? + (n, - 1)? 
nj tn, -— 2 


where y is f, which is the coefficient from the MMMC for the intervention effect. 


Note that in our analyses the outcome measure is standardized within the analytical 
sample (mean=0, standard deviation=1). As a result, we expected that g would likely be quite 
similar to the £, coefficient from the MMMC Model in the large samples we plan to collect later 
in the study. 


Though it is not standard to use g with a continuous treatment effect, as in the case of 
our dosage-dependent treatment measure, we have defined and matched populations n; and 
n, where treated students (n;) could have any positive treatment value<=1, and control 
students (n,) had a treatment value of zero. We could therefore calculate g in the case of our 
dosage-dependent treatment measure, and again expect that it would not differ substantially 
from the f, coefficient. It is crucial, however, to note that g and f, in the dosage-dependent 
models reflect the effect size projected for a student who would receive exposure to 
intervention teachers in all of their core classes. Along with this effect we report average 
dosage received by treated students so that the average treatment effect on treated students 
could be calculated. 
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